AD-A179  082 

INVESTIGATION  INTO  THE  USE  OF  NORMAL  AND  HALF-NORMAL 
PLOTS  FOR  INTERPRETI  <U)  ESSEX  CORP  ORLANDO  FL 

C  U  SIMON  25  MAR  87  EOTR-87-2  N61J39-85-D-8826 

1/2  ^ 

UNCLASSIFIED 

F/G  12/1 

ML 

AD-A179  082 


FINAL  REPORT 


J 


REPORT  NO.  EOTR*87-2 


OTIC  FILE  COPJf 


S 


a 

i 

i 


'K 

8 

h 


investigation  Into  the  Use  of  Normal  and  Half-Normal 
Plots  for  Interpreting  Results  from 
Screening  Experiments 


Charles  W.  Simon 

Essex  Corporation 
1040  Woodcock  Road 
Orlando,  Florida  32803 


* 


k . 


iy 


•<> 

¥  to 


UNCLASSIFIED 


tCURirr  CLASSIFICATION  OF  THIS  PAG 


n-Anqo? 


REPORT  DOCUMENTATION  PAGE 

14  Rt^®Mslf,CATION 

lb.  RESTRICTIVE  MARKINGS 

2a.  SECURITY  CLASSIFICATION  AUTHORITY 

3.  DISTRIBUTION / AVAILABILITY  OF  REPORT 

Approved  for  public  release- 

2b.  DECLASSIFICATION /DOWNGRADING  SCHEDULE 

distribution  unlimited 

4.  PERFORMING  ORGANIZATION  REPORT  NUM8ER(S) 

E0TR  87-2 

S.  MONITORING  ORGANIZATION  REPORT  NUMBER(S) 

6a.  NAME  OF  PERFORMING  ORGANIZATION 

Essex  Corporation 

6b.  OFFICE  SYMBOL 
(If  applicable) 

7a.  NAME  OF  MONITORING  ORGANIZATION 

Naval  Training  Systems  Center 

6c  AODRESS  (G'ty,  State,  and  ZIP  Code) 

1040  Woodcock  Road,  Suite  \ 
Orlando,  FL  32803 

127 

7b.  ADDRESS  {City.  State,  and  ZIP  Code) 

Orlando,  FL  32813-7100 

8a.  NAME  OF  FUNDING /SPONSORING 
ORGANIZATION 

Naval  Air  Systems  Command 

8b.  OFFICE  SYMBOL 
(If  applicable) ' 

9.  PROCUREMENT  INSTRUMENT  IDENTIFICATION  NUMBER 

N61339-85-D-0026 

10.  SOURCE  OF  FUNDING  NUMBERS 


PROGRAM 


PROJECT 

TASK 

NO. 

NO. 

8c  A  JRESS  (City,  State,  and  ZIP  Code) 

Washington,  DC  20361 


1 1 .  TITLE  (Include  Security  Classification) 

Investigation  into  the  Use  of  Normal  and  Half-Normal  Plots  for  Interpreting 
_ Results  from  Screening  Experiments 


12.  PERSONAL  AUTHOR(S) 

Charles  W.  Simon 


13a.  TYPE  OF  REPORT 

Final 


1ADATE  OF  REPORT  (Year.  Month.  Day)  IS.  PAGE  COUNT 

25  March  1987  152 


COSATI  COOES 


GROUP  |  SUB-GROUP 


18.  SUBJECT  TERMS  ( Continue  on  reverse  if  necessary  and  identify  by  block  number ) 

Normal  and  half-normal  plots.  Screening  experiments, 
R-Spi  Hover 


19.  ABSTRACT  ( Continue  on  reverse  if  necessary  and  identify  by  block  number) 

Daniel,  in  1959,  proposed  the  use  of  half-normal  probability  plots  as  a 
means  of  interpreting  the  results  from  unreplicated  "l  or  2V  v  factorial 
experiments.  Zahn,  in  1975,  suggested  modifications  to  Daniel’s  approach  and 
investigated  some  operating  characteristics  of  these  plots,  along  with 
techniques  for  estimating  population  sigma  from  the  slope  of  the  standardized 
contrasts  plotted  on  half-normal  probability  paper.  That  work  is  reviewed 
here. 


20  DISTRIBUTION /AVAILA8ILITY  OF  ABSTRACT  21.  ABSTRACT  SECURITY  CLASSIFICATION 

KKjnclassifiedajnlimiteo  □  same  as  rpt.  Dotic  users  UNCLASSIFIED  _ 


22a  NAME  OF  RESPONSIBLE  INDIVIDUAL  22b  TELEPHONE  (Include  Area  Code)  22c.  OFFICE  SYMBOL 

_ Dr.  Dennis  C.  Wiqhtman _  _  (305)  646-5130 _ N732 _ 


DO  FORM  1473,  84  MAR  83  APR  edition  may  be  u.ed  until  exhausted.  SECURITY  CLASSIFICATION  OF  THIS  PAGE 

All  other  edition,  are  ob.c’—  UNCLASS  I F I  ED 


*  *  i.  *  w>  . 


UNCLASSIFIED 


P  S6CUWITV  CLASSIFICATION  of  this  page 
19.  Abstract  (cont'd) 

J  This  project  examines  some  properties  and  the  usefulness  of  both  normal 
and  half-normal  probability  plots  when  applied  to  the  interpretation  of 
experimental  data  from  screening  experiments.  The  purpose  of  a  screening 
experiment  is  to  select  from  a  very  large  number  of  potentially  critical 
factors  the  ones  that  do  have  an  important  effect  on  the  performance  of  a 
particular  task,  so  that  these  may  be  investigated  in  depth.' - 

Characteristics  of  the  results  from  Monte  Carlo  simulations  of  2^ 
experiments,  where  f  =  5,  6,  and  7,  are  studied.  Artificial  data  from  5000 
runs  are  created  using  as  an  error  base  a  normal  population  with  a  mean  of 
zero  and  a  variance  of  one,  to  which  from  one  to  as  many  as  16  real  effects 
are  added  of  varying  sizes.  Ordered  contrasts  for  both  normal  and  half-normal 
plots  are  examined  along  with  the  "mislocation"  of  real  effects  at  ranks  other 
than  where  they  are  originally  located  prior  to  being  combined  with  the  error 
component.  The  relationship  between  number  and  sizes  of  real  effects  to 
percentage  of  real  effects  mislocated  is  determined. 

The  relative  effectiveness  of  normal  and  half-normal  plots  is  examined. 

The  advantages  and  disadvantages  of  each  are  discussed  in  the  context  of  the 
screening  experiment.  Examples  are  given  of  individual  normal  plots  to  show 
the  unreliability  of  this  graphic  technique  to  detect  marginal  effects. 
Guardrails  are  supplied  for  half-normal  plots  for  2 c  experiments,  where  k  = 

5  and  6.  These  guardrails  are  the  theoretical  sizes  that  contrasts  must  be  to 
be  called  "real"  at  some  probability  level.  The  pros  and  cons  regarding  the 
use  of  these  plots  are  discussed,  both  with  and  without  being  supplemented 
with  the  guardrails  to  facilitate  the  detection  of  real  effects. 

An  annotated  bibliography  is  supplied  of  19  published  papers  relevant  to 
this  project,  i.e.,  papers  that  describe  alternative  uses  for  probability 
plots  and/or  that  have  information  which  might  improve  the  conduct  of 
screening  experiments.  In  addition,  a  list  of  publications  are  given  which 
include  Daniel's  and  Zahn's  original  papers  on  half-normal  plots  in  their 
references.  Numerous  computer  programs  used  for  this  project  are  provided. 


UNCLASSIFIED 

SECURITY  CLASSIFICATION  OF  THIS  PAGE 


Uk 


TABLE  OP  CONTENTS 


Section 

ACKNOWLEDGEMENTS 


I  INTRODUCTION .  7 

Context .  7 

Economy .  8 

Multiplicity .  6 

Unreplicated  Experiments .  9 

Confounding . 1 . . . . .  9 

Data  Anomalies . 10 

Half-Normal  Plots . 11 

II  REVIEW  OP  DANIEL'S  AND  ZAHN'S  PAPERS .  13 

Daniel's  Papers . 13 

Daniel's  Conclusions .  15 

Zahn's  Papers .  16 

Zahn's  Procedures  for  Estimating  a .  19 

Zahn's  Investigations . 21 

Zahn's  Conclusions .  21 

III  RATIONALE  FOR  THE  PRESENT  PROJECT . 25 

IV  CONSTRUCTING  NORMAL  AND  HALF-NORMAL  PLOTS .  29 

Constructing  Normal  Plots .  29 

Constructing  Half-Normal  Plots .  31 

Standardized  Plots .  32 

V  THE  BASIC  SIMULATION .  33 

Constructing  the  Data  Base . 33 

Boundaries  of  the  Experimental  Space  Examined 

in  this  Project .  35 

Example . 35 

Accuracy..... . 38 

Relating  Zahn's  Parameters  to  Those  Used  in 

the  Present  Investigation .  38 


VI  ANALYSES  AND  RESULTS . 

Characteristics  of  Normal  Plot  Data . 

Effect  of  Marginal  Size  Effects . 

Effect  of  Having  Mixed  Postive  and  Negative  Effects.. 

Effect  of  Mixing  the  Sizes  of  the  Real  Effect . 

Effect  of  a  Different  Number  of  Real  Effects . 

Effect  of  Increasing  the  Size  of  the  Experiment . 

Summary  of  Normal  Plot  Characteristics  as 

a  Function  of  N,  K,  R,  R+,  and  D . 


41 

41 


56 

56 


63 


1 


Availability  Codes 


diui/or 

L^ejciai 


V.V.- v.  •v,-'.-’.-'-  v-V\  V  V  •. 


v'LvvvV/.'i' 


Drier 


TABLE  OP  CONTENTS  (continued) 


Section 


Page 


VI  (continued) 

Normal  Versus  Half-Normal  Plot  Data .  65 

R-Sp  lllover .  65 

Estimating  Sigma .  73 

Over-  and  Underestimating  the  Number  of  Error 

Contrasts . 73 

Single  Run  Data . 75 

Detection . 75 

Estimating  Sigma .  87 

Guardrails . 93 

VII  RELEVANT  PAPERS . 97 

Identifying  Significant  Effects  in  Unreplicated 

Experiments .  98 

Other  Applications  for  Plots . 99 

Variations  on  the  Normal  Plot,  Graphic  and  Otherwise . 101 

Graphical  Techniques  in  Statistical  Data  Analysis .  103 

Multiple  Comparison  Procedures .  105 

VIII  CONCLUSIONS .  107 

Additional  Efforts .  110 

REFERENCES .  Ill 

APPENDIX  A  -  LITERATURE  SEARCH .  113 

APPENDIX  B  -  EXPECTED  VALUES  OF  NORMAL  AND  HALF-NORMAL 

ORDER  STATISTICS .  119 

APPENDIX  C  -  PROGRAM  FOR  GENERATING  DATA  BASES  AND 
FOR  PERFORMING  SUBSEQUENT  ANALYSES 

USED  IN  THIS  REPORT .  123 

APPENDIX  D  -  COMPUTER  PROGRAM  FOR  GENERATING  EXPECTED 
VALUES  OF  NORMAL  AND  HALF-NORMAL  ORDER 

STATISTICS .  137 

APPENDIX  E  -  DEVIATION  OF  STANDARD  ERROR  OF  THE 

CONTRASTS .  139 

APPENDIX  P  ~  PREDICTING  R-SPILLOVER  USING  CONTRASTS 

AND  CONTRAST  STANDARD  DEVIATIONS .  141 

APPENDIX  G  -  PROGRAM  FOR  GENERATING  GUARDRAILS .  145 

NOTATIONS  AND  TERMINOLOGY .  149 


2 


Mafwa  Ji  >'»  Wl  ■  » 


LIST  OF  ILLUSTRATIONS 


Figure 


1  Example  of  Half-Normal  Plot .  12 

2  Example  of  Normal  Plot .  30 

3  Example  of  Half-Normal  Plot .  31 

4  Relating  Number  of  Real  Effects,  Size  of  Real  Effects, 

and  R-Spillover  in  First  Intended  Error  Rank .  64 

5-a  Sample  Individual  Normal  Plot  for  Situation: 

n-32,  k-31,  r-8+,  d  -  1.000 .  77 

5-b  Sample  Individual  Normal  Plot  for  Situation: 

n-32,  k-31,  r-8+,  d  -  l.OOo .  78 

5-c  Sample  Individual  Normal  Plot  for  Situation: 

n-32,  k-31,  r-8+,  d  -  l.OOo .  79 

5- d  Sample  Individual  Normal  Plot  for  Situation: 

n-32,  k-31,  r«8+,  d  -  l.OOo .  80 

6- a  Sample  Individual  Normal  Plot  for  situation: 

n=32,  k-31,  r-8+,  d  =  1.25o .  81 

6-b  Sample  Individual  Normal  Plot  for  Situation: 

n-32,  k-31,  r-8+,  d  =  1.250 .  82 

6- c  Sample  Individual  Normal  Plot  for  Situation: 

n-32,  k-31,  r-8+,  d  -  1.250 .  83 

7 - a  Sample  Individual  Normal  Plot  for  Situation: 

n-32,  k-31,  r-5+,3-,  d  -  1.25o .  84 

7-b  sample  Individual  Normal  Plot  for  Situation: 

n-32,  k-31.  r-5+,3-,  d  -  1.250 .  85 

7- c  Sample  Individual  Normal  Plot  for  Situation: 

n-32.  k-31.  r-5+,3-,  d  -  1.25o .  86 

8- a  Sample  Individual  Plots  for  Situation: 

n-32,  k-31.  r-8+,  d-1.67o .  88 

8-b  Individual  Plots  for  Situation: 

n-32,  k-31,  r-8+,  d»1.67o... .  89 

8-c  Sample  Individual  Plots  for  Situation: 

n-32,  k-31,  r-8+,  d-1.67o . . .  90 

9  Histogram  of  50  Population  Sigmas 

Situation:  n-32,  k-31,  n-8+.  d=l . 25o . .  91 


Cv/A-.v.. 


LIST  OF  TABLES 


Table 


1  Empirical  Behavior  of  Version  S  in  2-  and  4-Situations 

with  Real  Contrasts  of  Sizes  2"o",  4"o",  and 

6"o"  Present,  Using  a  *  0.05,  0.20,  and  0.40 .  24 

2  Example  of  Analyses:  Contrast,  Number  of  Real  Effects, 

Standardized  Contrast,  and  Contrast  Standard 

Deviation  at  Each  Rank . .  36 

3  Example  of  Original  and  Standardized  Ordered  Contrasts 

*Ri,k'  an<*  Contrast  Standard  Deviations.... .  43 

4  Expected  Values  of  Normal  Order  Statistics  for 

n  *  23  and  n  ■  119 .  44 

5  Changes  in  Results  as  a  Function  of  the 

Magnitude  of  Real  Effects .  46 

6  Spillover  of  Real  and  Error  Contrasts  as  a  Function 

of  Effect  Size . 50 

7  Results  with  Mixed  Positive  and  Negative  Effects .  52 

8  Results  Where  the  Sizes  of  the  Real 

Effects  are  Mixed .  55 

9  Changes  in  Results  as  a  Function  of  the 

Number  of  Real  Effects .  57 

10  Standardized  Contrast  Degradation  and  R-Spillover  in 

First  Error  Rank  as  a  Function  of  n  Where  r  -  8 

and  d#  »  +1.000  .  61 

11  Standardized  contrast  Degradation  and  R-Spillover 

in  First  Error  Rank  as  a  Function  of  n  Where 
r  =  8  and  d,  =  +1.000 . 63 

12  Half-Normal  Plot  Data .  66 

13  Comparing  R-Spillover  of  Normal  and  Half-Normal  Plots  on 

Population  sigma  Estimations  When  the  Number  of  Error 
Contrasts  are  Over-  and  Underestimated .  72 

14  Comparing  the  Robustness  of  Normal  and  Half-Normal 

Plots  on  Population  Sigma  Estimations  When  the 
Number  of  Error  Contrasts  are  Over-  and 

Underestimated .  74 

15  Effect  of  Errors  in  the  Initial  Estimate  of 

Sigma  on  the  Final  Estimate  of  Sigma .  92 

16  Guardrails  for  Half-Normal  Plots  Where 

k  -  31  and  63 .  94 


s- 


6 


‘.V 


ACKNOWLEDGEMENTS 


The  help  provided  on  this  project  by  the  following  persons  Is  gratefully 
acknowledged: 

Dr.  David  G.  Weinman,  Department  of  Statistics,  Hollins  College,  Virginia, 
a  consultant  on  this  project,  who  patiently  provided  explanations  of 
statistical  concepts  and  offered  constructive  advice  regarding  the  editing  of 
the  report. 

Dr.  Daniel  P.  Westra,  Essex  Corporation,  Orlando,  who  originally  proposed 
this  project  and  who  directed  and  wrote  some  of  the  computer  programs  used  for 
this  project. 

Ronald  Mauk,  who  helped  write  the  computer  programs  and  ran  many  analyses. 

Patrick  Ellis,  who  ran  numerous  data  analyses. 

Patricia  Patrizzi,  who  managed  the  preparation  of  this  report. 


SECTION  I 


INTRODUCTION 

The  purpose  of  this  project  was  to  Investigate  the  operating 
characteristics  of  normal  and  half-normal  plots  and  to  evaluate  their 
effectiveness  when  used  to  interpret  data  derived  from  screening  experiments. 

CONTEXT 

Modern  industrial  and  military  equipments  are  frequently  quite  complex 
with  interacting  elements  that  may  affect  operator  performance  in 
unforeseeable  ways.  For  several  decades,  efforts  to  optimize  the  design  of 
such  equipment  have  depended  in  part  on  the  results  from  experiments  in  which 
representative  operators  are  placed  in  representative  situations  and  are 
required  to  perform  representative  tasks  on  the  different  equipment 
configurations  in  order  to  determine  which  configuration  results  in  the  best 
cost-effective  performance. 

While  recognizing  that  a  relatively  large  number  of  factors  related  to  the 
operator  and  the  environment,  as  well  as  the  equipment  itself,  affect 
performance  in  any  given  task,  investigators  have  carefully  avoided  studying 
more  than  a  few  equipment  factors  —  usually  less  than  four  —  at  a  time  in 
any  single  experiment  (Simon,  1976b).  To  cover  a  more  expansive  space,  a 
series  of  few-factor  experiments  from  different  parts  of  the  space  are  usually 
run.  This  spotted  approach,  however,  quickly  becomes  economically  unfeasible 
and  never  adequately  covers  the  multifactor  space  of  interest  nor  obtains 
sufficient  information  regarding  factor  interactions.  Consequently,  the 
information  needed  to  properly  design  equipment  in  which  10  or  20  or  more 
factors  may  have  an  impact  on  engineering  decisions  and  operator  performance 
is  never  obtained. 

Simon  (1973,  1977a,  1977b)  proposed  a  holistic  methodology  whereby  a  great 
many  factors  could  be  investigated  economically  in  a  single  human  performance 
experiment.  Its  application  to  naval  training-simulator  equipment  was 


proposed  (Simon,  1979)  and  successfully  employed  (Vestra,  Simon  et  al.,  1981; 
Westra,  1982), 

ECONOMY 

This  holistic  approach  makes  many-factor  experiments  economically  feasible 
by  collecting  the  data  sequentially  in  small  blocks.  Rather  than  attempt  to 
do  this  with  a  mosaic  of  small  experiments  located  at  different  parts  of  the 
experimental  space,  the  holistic  approach  collects  the  data  with  minimum 
precision  over  the  total  space  and  later  improves  the  precision  where  it  is 
needed.  Initially,  all  of  the  potentially  critical  factors  are  studied  in  a 
screening  design  capable  of  defining  the  simplest  model  of  the  experimental 
world.  While  not  precise,  this  early  overview  will  reveal  which  factors  and 
what  segments  of  the  total  space  are  critical,  usually  a  small  fraction  of  the 
original  effort.  This  screening  phase  reduces  the  data  needed  to  bring  the 
precision  of  the  information  to  the  required  level.  By  testing  after  each 
block  and  collecting  no  more  data  once  the  experimental  results  adequately 
model  the  real  world,  considerable  savings  can  be  realized  since  a  second-  or 
third-order  model  is  usually  enough  to  approximate  for  all  practical  purposes 
the  performance  on  a  great  many  human  performance  tasks. 

The  interpretation  of  the  results  from  a  screening  experiment  requires  a 
delicate  balance  between  two  opposing  goals  affecting  the  conduct  of  the 
future  research.  On  the  one  hand,  the  investigator  wants  to  reduce  the  number 
of  candidate  factors  to  a  small  enough  number  to  make  the  subsequent  data 
collection  feasible.  On  the  other  hand,  he  wants  to  be  certain  that  all 
factors  likely  to  be  critical  in  the  task  under  investigation  will  be 
identified.  When  one  must  categorize  a  factor  as  critical  or  not  on  the  basis 
of  the  screening  data,  the  difficult  decisions  rest  with  the  effects  of 
marginal  size. 

MULTIPLICITY 

When  a  great  many  constrasts  are  being  evaluated  for  significance  at  the 
same  time  --  31  or  63  are  the  numbers  frequently  found  in  screening 
experiments  --  some  effects  can  appear  several  times  larger  than  the  average 


even  when  no  effect  is  real.  For  example,  in  an  experiment  with  31  contrasts, 
the  size  of  the  largest  contrast  can  be  2.4  times  larger  than  the  average 
contrast  purely  by  chance  alone.  Using  the  traditional  0.05  significance 
level  in  such  an  experiment  would  cause  unreal  effects  to  be  judged  real  in 
over  half  of  all  experiments  being  done  (Daniel,  1959,  p.  312). 

Most  human  performance  investigators  tend  to  overlook  this  problem  of 
multiplicity  when  they  use  the  conventional  F-test  of  significance  to  evaluate 
the  data  from  an  analysis  of  variance.  This  neglect  can  lead  to  some  serious 
misinterpretations  unless  the  proper  corrections  are  made  in  the  allowable 
error  rate  of  the  individual  comparisons  as  well  as  that  of  the  overall  family 
of  contrasts. 

UNREPLICATED  EXPERIMENTS 

As  the  number  of  factors  to  be  studied  increases,  the  amount  of  data  that 
must  be  collected  also  increases.  Large  mult  if actor  experiments  are 
economically  feasible  only  because  the  sequential  application  of  fractional- 
factorial  designs  allows  the  size  of  the  screening  experiment  to  increase  at  a 
far  slower  rate  than  the  number  of  factors  being  investigated. 

However,  even  when  fractional-factorials  of  the  appropriate  resolution  are 
used,  the  amount  of  data  which  must  be  collected  can  still  reach  impractical 
limits.  Logistical  conditions  as  well  as  time  limitations  may  be  operating, 
for  example,  the  availability  of  experimental  subjects  (operators)  and  of  the 
equipment  and  supporting  staff.  For  that  reason,  when  31  to  64  experimental 
conditions  are  involved,  the  traditional  luxury  of  replicating  the  experiment 
may  not  be  possible.  Consequently,  this  unreplicated  screening  experiment 
will  provide  no  direct  estimate  of  the  error  variance  to  serve  as  the 
denominator  of  the  F-test  used  to  evaluate  the  significance  of  the 
experimental  effects. 


CONFOUNDING 

In  the  unreplicated  experiment,  therefore,  error  variance  traditionally  is 
estimated  by  other  methods,  namely:  (1)  Obtaining  it  from  the  results  of 


similar  experiments  done  at  another  time,  and  (2)  Pooling  the  variance  of 
higher-order  Interaction  effects  on  the  a  priori  assumption  that  they  are  not. 
likely  to  be  significant. 

Neither  technique  is  really  justified  in  human  performance  research.  For 
one  thing,  because  of  poor  subject  sampling  procedures  and  of  the  large  number 
of  uncontrolled  or  unidentified  factors  that  may  differentially  affect 
different  experiments  which  may  appear  superficially  the  same,  using  the  error 
variance  from  another  experiment  is  risky  business.  For  another,  higher-order 
interactions  —  the  term  usually  refers  to  effects  that  include  third-order  or 
higher  interactions  —  may  sometime  be  significant.  This  is  particularly  true 
in  human  performance  experiments  in  which  equipment  and/or  performance 
measurements  are  improperly  scaled. 

In  screening  experiments,  the  focus  of  this  report,  highly  saturated 
fractional-factorials  will  be  employed.  Higher-order  interaction  effects, 
although  expected  to  be  insignificant,  are  generally  confounded  with  the 
effects  of  interest  and  therefore  unavailable  for  estimating  error.  While 
subsequent  data  collection  will  unconfound  the  critical  effects,  for  the 
screening  phase,  some  alternative  way  of  estimating  the  error  variance  must  be 
found  when  an  unreplicated,  fractional-factorial  design  is  employed. 

DATA  ANOMALIES 

When  data  are  sparsely  drawn  from  coordinates  over  a  larger  experimental 
space  (the  screening  experiment),  when  the  performance  of  human  operators  is 
involved,  and  when  the  tasks  are  performed  on  complex  equipment  which  may 
occasionally  break  down,  irregularities  in  the  experimental  data  may  be 
expected,  without  a  dedicated  effort  to  do  so,  these  may  not  be  recognized 
when  they  occur  nor  ever  discovered  after  the  data  has  been  fed  into  a 
computer  and  regurgitated  as  an  F-table  of  an  analysis  of  variance. 

Conclusions  drawn  from  such  distorted  data  therefore  are  likely  to  be 
erroneous  unless  the  investigator  finds  ways  of  discovering  the  abnormalities. 


HALF-NORMAL  PLOTS 


The  problems  cited  above  are  not  unique  to  screening  experiments  and  must 
be  attended  to.  In  large  multifactor  screening  studies,  the  sequential 
approach  should  handle  any  overconcern  with  the  confounding  of  effects  during 
the  early  stages  of  the  investigation.  But  there  remain  the  problems  of  data 
interpretation  brought  about  by  the  absence  of  an  external  estimate  of  error, 
the  confusion  from  multiplicity,  and  the  distortion  from  outliers. 

Half-normal  plots,  when  introduced,  seemed  to  help  reduce,  if  not  solve,  these 
problems.  In  addition,  this  graphic  presentation  provides  the  investigator 
with  a  quick  overview  of  his  results,  facilitating  interpretation  beyond  any 
statistics. 

Daniel  (1956,  1959)  is  most  frequently  credited  as  being  the  one  who 
popularized  the  use  of  the  half-normal  plot  to  interpret  the  results  from 
unreplicated  2^  and  2^  P  factorial  experiments  and  to  search  for  anomalies 
in  the  data.  These  balanced  designs  are  the  ones  most  commonly  used  in 
screening  studies.  An  example  of  how  the  imaginary  results  from  an  experiment 
using  a  2 4  design  would  appear  on  a  half-normal  plot  is  shown  in  Figure  1. 

This  half-normal  plot  shows  the  empirical  cumulative  distribution  of  the 
set  of  15  orthogonal,  unsigned  contrasts  when  plotted  onto  a  special  grid  with 
the  contrasts  plotted  on  the  abcissa  and  the  positions  of  the  expected  values 
of  order  statistics  for  a  half-normal  distribution  for  15  cases  plotted  on  the 
ordinate.  If  no  real  effects  are  present  in  this  data,  the  plots  differing 
solely  by  chance  would  tend  to  fall  along  a  straight  line.  In  our  example, 
however,  the  five  largest  points  deviate  sufficiently  from  the  line  formed  by 
the  remaining  points.  One  would  conclude  that  those  five  effects  are  probably 
real  and  that  the  remaining  points  are  made  up  of  error  contrasts. 

Daniel  (1959)  proposed  the  use  of  standardized  half-normal  plots.  The  raw 

contrasts,  x  ,  are  standardized,  x  ,  by  dividing  each  by  an  initial 
x  1 

estimate  of  the  standard  error  of  the  contrasts.  He  then  plotted  "guardrails" 
on  his  grid,  i.e.,  lines  connecting  critical  values  representing  the  amount  a 
contrast  that  might  exceed  the  central  slope  of  the  null  experiment  purely  by 
chance  by  some  probable  amount. 


Figure  1.  Example  of  Half-Normal  Plot  for  a  2* 

Factorial  Experiment 

Daniel  (1959)  also  showed  how  the  ordered  data  of  the  half-normal  plot 
might  provide  clues  to  abnormalities  in  the  data,  detecting  such  things  as  bad 
values,  heteroscedasticity,  and  defective  randomization.  Later,  Daniel  (1976) 
suggested  that  the  full-normal  plot,  in  which  the  signs  of  the  contrasts  are 
retained,  would  provide  more  information  about  the  data  characteristics  than 
the  half-normal  plot  of  the  absolute  contrasts. 

Zahn  (1975a,  b)  suggested  changes  in  Daniel's  plotting  procedures.  He 

also  proposed  to  use  the  slope,  SL,  of  the  plotted  error  contrasts  on  this 

standardized  grid  to  provide  a  final  internal  estimate  the  standard  error, 

s  ,  from  which  an  estimate,  s,  of  the  population  sigma  can  be  derived.  In  a 
c 

series  of  Monte  Carlo  studies,  he  compared  several  methods  of  estimating  the 
slope . 
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SECTION  II 


REVIEW  OF  DANIEL’S  AND  ZAHN’S  PAPERS 

Daniel’s  and  Zahn's  papers  on  half-normal  plots  are  reviewed  here  briefly 
as  background  for  the  present  effort.  While  Hazen  (1914)  suggested  the 
principle  of  linearizing  the  normal  distribution  in  a  study  of  floods,  in 
recent  years,  Daniel's  work  is  generally  credited  for  popularizing  and 
expanding  the  applications  for  normal  and  half-normal  plots. 


The  extent  to  which  people  have  found  the  half-normal  plot  to  be  useful  is 
reflected  in  the  number  of  papers  between  1959  and  the  present  that  referenced 
Daniel's  work  (Appendix  A).  In  the  discussion  below,  Daniel’s  procedures  are 
described  as  he  did  them,  although  subsequently  Zahn  suggested  changes. 

DANIEL'S  PAPER 

Daniel  (1959)  proposed  the  use  of  the  half-normal  plot  as  a  graphic  tool 
for  evaluating  and  interpreting  certain  classes  of  experimental  data.  He 
noted  that  bad  values,  heteroscedasticity,  dependence  of  variance  on  means, 
and  some  types  of  defective  randomization  have  characteristic  forms  on  the 
olots.  He  also  suggested  that  the  half-normal  plot  might  be  used  in  an 
unreplicated  experiment  to  estimate  the  population  a,  where  a2  equals 
the  variance  of  the  data,  and  to  use  that  information  to  identify  which 
effects  in  a  2^  or  2^  ^  experiment  were  significant. 

Daniel  modified  normal  probability  paper  to  fit  the  half-normal  case  by 
deleting  the  given  normal  probability  scale,  P,  and  replacing  it  with  one 
suitable  for  the  half-normal  case,  where  P'  =  2P  -  100.  After  marking  off  the 
ordinate  scale  with  those  revised  values,  he  selected  specific  P'  values  as 
the  plotting  positions  for  k  ranks  using  this  equation: 


P '  =  (i  -  0.5)  f  k ;  i  =  1,  2,  3...  k. 


Daniel  plotted  the  unsigned  contrasts  on  the  grid’s  abscissa. 


Daniel  noted  that  if  the  experiment  were  a  null  one  —  that  is,  no  factor 

had  a  real  effect  —  s  can  be  roughly  estimated  by  the  contrast,  x  in  k 

c  a 

contrasts,  where 


a  =  Integer  nearest  x  (2) 

O.CI3K 

For  k  =  31,  63,  and  127,  the  s  would  be  the  contrasts  at  ranks  22,  44, 

c 

and  86  respectively  on  the  ordered  contrast  scale.  However,  if  some  effects 
are  real,  they  must  be  removed  before  Equation  2  would  be  used. 


Daniel  suggested  that  by  standardizing  the  half-normal  plot,  fixed 
"guardrails"  could  be  drawn  at  different  a  values,  when  a  is  the 
probability  of  calling  an  observed  contrast  real  when  in  fact  it  is  not.  He 
standardized  his  contrasts,  x^,  by  dividing  each  one  by  the  "best"  estimate 
of  the  standard  error  of  the  contrasts,  x  ,  in  a  null  experiment. 

3i 


Daniel  used  as  his  test  statistic  (on  which  the  location  of  a  guardrail 
was  based): 


x  ,k 


a,k 


(3) 


where: 


x  is  the  largest  in  absolute  magnitude  of  the  k  contrasts  from  a 

1  r  1C 

random  normal  variable  with  a  population  mean  of  zero  and 
variance  o  2 . 


x 

a,k 


is  the  contrast  at  the  rank  determined  by  Equation  2,  serving  as 
an  initial  estimate  of  a. 


Co) 

Daniel  (and  later  Zahn)  noted  that  the  distribution  of  log  1Qtk  "is 
quite  closely  approximated  by  a  normal  distribution"  (p.  319).  When  the 
t-statistic  is  plotted  on  a  normal  probability  scale  it  appears  as  a  straight 
line. 


Daniel  drew  a  guardrail,  for  example,  to  yield  an  error  rate  of  a=0.05 
false  positives  per  experiment  by  drawing  a  line  through  the  0.05  critical 
values  for  the  statistics  t  ,  t  ,  t  ,  and  t 
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Daniel  proposed  that  guardrails  representing  a  rather  large  error  rate 
(i.e.,  false  positive)  be  used,  e.g.,  an  a  of  0.40  or  even  0.80,  "knowing 
that  this  will  produce  a  number  of  'false  alarms’.  The  justif ication  is  of 

course  in  the  increased  sensitivity  to  small  effects _  Erroneously  selected 

effects  have  good  prospects  of  being  exposed  in  later  work.  Missed  real 
effects  on  the  other  hand  are  likely  to  be  dropped  from  study,  and  if  they  are 
numerous,  loss  of  knowledge  of  their  combined  effects  may  be  serious." 

(Daniel,  1959,  p.  322-325). 

DANIEL'S  CONCLUSIONS.  Daniel  concluded  that  "when  only  a  small  proportion  of 
the  totality  of  contrasts  have  effects,  this  plot  can  be  used  to  make 
judgments  about  the  reality  of  the  largest  effects  found"  (p.  339). 

In  general,  effects  or  outliers  have  to  be  rather  large,  i.e.,  more  than 
four  times  larger  than  x  (the  estimated  a)  to  be  clearly  visible,  single 

cl 

outliers  may  frequently  be  detected  from  the  plots  when  the  smaller  values, 
presumably  error  contrasts,  do  not  point  toward  the  origin.  Corrective 
techniques  were  suggested  (p.  331). 

Daniel  showed  how  multiple  plots  can  improve  the  interpretation  of  split 
plot  designs  (p.  329).  This  is  accomplished  by  plotting  separately  the 
positive  and  negative  halves  of  the  effect  used  to  block  on,  done  of  course  on 
a  grid  suited  to  the  smaller  number  of  contrasts  per  plot.  The  contrast  used 
in  blocking  is  omitted.  One  might  expect  different  error  variances  for  the 
two  plots,  a  fact  that  is  hidden  in  the  combined  plot  and  one  that  could 
distort  the  interpretation  of  the  results. 

Daniel  also  examined  how  variations  in  the  plot  pattern  might  be  used  to 
warn  of  deviations  from  normality  in  the  data  (p.  331-338).  However,  he 
pointed  out  that  certain  unusual  patterns  might  have  multiple  causes.  No 
systematic  investigation  of  this  process  was  pursued  in  his  paper. 


It  is  only  fair  to  add  that  Daniel  recognized  that  his  paper  was  not 
definitive  and  that  a  better  understanding  of  how  and  what  he  had  proposed  was 
still  needed.  He  also  emphasized  the  subjective  basis  of  this  graphic 
technique  and  warned  that  using  it  in  a  routine  way  as  a  substitute  for  an 
anslysis  of  variance  might  be  "catastrophic"  (p.  338).  He  noted  that  on  an 
individual  basis,  there  can  be  wide  variations  in  the  appearance  of  the  plots, 
even  when  no  effects  were  introduced  into  the  original  data.  He  raised 
questions  regarding  the  best  method  of  estimating  the  standard  error  of  the 
contrasts  and  wondered  whether  the  2^  ^  experiment  should  be  at  least 
partially  duplicated.  He  pointed  out  the  importance  of  not  relying  solely  on 
this  graphic  technique.  He  emphasized  the  dangers  of  a  univariate  (referring 
to  the  response)  rather  than  a  multivariate  approach  to  research  (p.  339). 

ZAHN’S  PAPERS 

Zahn  (1975a,  b)  wrote  two  papers  on  the  construction  and  operating 
characteristics  of  the  half-normal  plot.  In  the  first  paper,  he  proposed  a 
change  in  the  way  in  which  Daniel  prepared  the  half-normal  grid  and  exposed  a 
major  flaw  in  the  way  Daniel  selected  the  critical  values  for  his  guardrails. 
In  the  second  paper,  Zahn  evaluated  several  ways  of  estimating  the  error 
variance  (based  on  the  slope  of  standardized  error  contrasts)  from  the  data  of 
an  unreplicated  experiment. 

Zahn  (1975a)  suggested  reversing  the  scales  that  Daniel  plotted  on  the 
ordinate  and  abcissa  of  the  plotting  grid,  to  make  the  half-normal  plot 
correspond  more  closely  to  the  usual  regression  analysis  graph  on  which  the 
random  variable  is  plotted  as  the  ordinate"  (p.  191).  The  half-normal 
percentiles  were  plotted  on  the  horizontal  axis.  While  configuring  the  grid 
either  way  would  provide  the  same  information,  noticing  which  is  being  used 
before  interpreting  the  plot  patterns  is  important  since  real  effects  deviate 
from  the  null  line  in  opposite  directions  in  the  two  cases.  Failing  to 
perceive  this  difference  could  lead  to  serious  errors  in  interpretation. 


Zahn  showed  how  Daniel  erred  In  his  calculations  of  the  guardrails.  The 

t  =  x./x  ,  1=1,  2,...,  n  which  Daniel  used  are  not  half-normally 
3.  1  d 

distributed  although  he  plotted  them  against  percentiles  of  the  standard 
half-normal  distribution  (p.  192).  This  discrepancy,  however,  proves  to  be 
small  after  the  number  of  plotting  positions  exceed  20  (p.  193),  the  case  of 
an  experiment  of  size  2  where  (f-p)  equals  five.  On  the  standardized 
grid,  drawing  a  line  from  the  original  through  the  value  at  the  rank  nearest 
to  the  0.683  probability  value  will  represent  the  expected  values  at  all  ranks 
in  a  null  experiment. 

Zahn  noted  a  more  serious  error  in  the  calculations  Daniel  used  to  plot 
the  position  of  his  guardrails  and  the  error  rate  they  are  intended  to  control 
(p.  195).  Daniel,  in  calculating  his  t-statistic,  failed  to  change  the  sample 
size  in  x  for  each  rank,  which  resulted  in  erroneous  tables  from  which  his 

d 

critical  values  were  selected. 

The  choice  of  critical  values  depends  on  what  aspect  of  false  positive 
behavior  one  is  trying  to  control.  Zahn  wished  to  control  the  probability  of 
a  non-zero  family  error  rate,  the  PER.  This  is  the  probability  of  obtaining 
at  least  one  false  positive  in  the  family  of  contrasts  when  no  real  contrasts 
are  present.  In  the  null  situation,  each  test  assumes  that  no  effects  are 
present,  and  yields  at  least  one  false  positive  if  and  only  if  the  largest 
contrast  is  declared  significant.  This  occurs  when  the  test  statistic  for 
examining  the  largest  contrast  is  larger  than  its  a  level  critical  value. 

In  that  case,  under  the  hypothesis  that  all  contrasts  equal  zero,  PER  equals 
a  (p.  195). 

Zahn  proposed  a  theorem  with  limited  proof,  namely,  that  the  "half-normal 
plot  using  a-level  critical  values  has  PER  equal  to  or  less  than  a), 
regardless  of  how  many  real  contrasts  of  various  sizes  are  present  in  the 
experiment  being  analyzed  (p.  195).  The  equation  for  the  probability  error 
rate  for  e  independent  tasks  is: 


This  equation  Is  not  completely  accurate  when  applied  to  ordered  data,  for  In 
that  case,  the  contrasts  are  not  Independent,  although  Equation  4  assumes  they 
are. 


In  practice,  however.  It  may  be  wise  to  set  the  guardrail  with  an  a 
value  equal  to  a  PER  for  (e  +  1)  contrasts  as  calculated  for  the  F-test.  Zahn 
(p.  198)  cites  as  an  example  a  study  In  which  an  F-test  was  used  to  determine 
whether  or  not  four  effects  were  real  (out  of  the  nine  being  examined)  at  a 
0.05  significance  level.  The  F-test  In  that  case  had  an  approximate  PER  of 
0.37.  On  a  half-normal  plot,  the  same  effects  were  declared  significant  when 
the  0.40  guardrail  was  used  (p.  198). 

Zahn's  guardrails  differ  from  Daniel's  because  Daniel  miscalculated  his 
t-statistlc,  t^.  The  significance  of  any  contrast  depends  not  only  on  its 
distance  from  the  expected  contrast  but  also  its  rank.  Thus  the  t-statlstic, 
to  be  calculated  using  Equation  3,  requires  that  the  numerator  and  denominator 
in  the  t-statistlc  be  reduced  one  rank  each  time  a  contrast  is  declared 
significant  In  order  to  test  the  next  lower  contrast. 

Because  Zahn  generated  his  error  contrasts  directly,  when  he  refers  to  the 
size  of  a  real  effect  In  terms  of  a  —  a  symbol  used  in  this  paper  to 
represent  the  population  a  —  he  is  using  it  to  represent  the  standard  error 
of  contrasts.  The  reader  should  be  aware  that  the  standard  error  of  contrasts 
equals  the  population  sigma  times  the  square  root  of  four  over  n.  This  fact 
does  not  affect  the  comparisons  he  made  In  his  study.  It  can  matter,  however, 
when  we  wish  to  put  Zahn's  results  in  a  broader  context  and  compare  them  with 
the  results  In  the  Investigation  for  this  project.  Similarly,  the  slopes  of 
his  error  contrasts  actually  approximate  the  standard  error  of  the  contrasts 
although  he  refers  to  them  as  a.  From  this  point,  when  we  refer  to  Zahn's 
o's,  we  shall  put  them  in  quotation  marks,  thus  "o“,  to  remind  us  that  it 
is  really  the  standard  error  of  the  contrast. 

Half-normal  plots  may  be  used  to:  (1)  Detect  real  effects,  and  (2) 

Estimate  the  error  variance.  Zahn  compared  Daniel's  procedure  for  using  the 
half  normal  plot,  referred  to  as  Version  0,  against  several  versions  of  his 
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own.  Whereas  Daniel  had  used  a  single  number  to  estimate  "a",  i.e.,  the 
contrast  falling  at  the  rank  nearest  the  0.683  percentile  of  the  ordered 
contrasts,  Zahn,  In  order  to  obtain  a  more  stable  and  accurate  estimate, 
combined  selected  error  contrasts  to  calculate  the  slope  of  the  regression 
line  of  the  plots  on  a  standardized  half-normal  grid.  This  slope  is  an 
estimate  of  "o." 

ZAHN'S  PROCEDURES  FOR  ESTIMATING  o.  Zahn's  Version  x'  procedure  is  this: 

After  obtaining  the  ranked  unsigned  contrasts,  he  plots  them  at  the 

appropriate  plotting  positions  of  a  standardized  half-normal  grid.  He  still 

uses  the  rank  nearest  the  0.683  percentile  as  his  initial  estimate  of  "o"  to 

standardize  the  scale  of  his  vertical  axis.  The  critical  values  for  the 

guardrails  are  determined  and  plotted.  Then  starting  with  the  largest 

contrast,  he  determines  whether  it  is  plotted  beyond  the  guardrail.  If  so,  he 

declares  it  significant  and  looks  at  the  next  largest,  and  so  forth  until  a 

contrast  is  plotted  within  the  guardrail  or  "until  x  is  encountered"  (p. 

n-r 

202).  The  remaining,  smaller  contrasts  are  all  declared  insignificant  and  are 

called  the  error  contrasts.  He  replots  these  error  contrasts  on  a  new 

half-normal  grid  against  the  appropriate  expected  values  of  normal  order 

statistics,  z.  where  i  is  the  ith  rank  and  e  is  the  number  of  error 
X  #  6 

contrasts.  He  next  computes  sf,  the  final  estimate  of  "a",  where  sf  is 
the  slope  of  the  ordinary  least  squares  regression  line  through  the  origin  of 
x  on  z,  fitted  to  the  smallest  m  =  [0.7(e+l>]  of  the  e  error  contrasts.  The 
equation  for  this  final  estimate  of  "a"  is: 


(5) 


where : 


m  =  [0.7  (e  +  1)  ]  . 


The  0.7  value,  the  proportion  of  the  error  contrasts  (plus  one)  that  were  used 
to  estimate  the  slope,  was  empirically  selected  in  Zahn's  (1975b)  second  paper 
as  being  "a  reasonable  choice"  (p.  206-208). 

Zahn  investigated  several  other  versions  which  he  felt  might  improve  the 
detection  process.  In  his  Version  S.  instead  of  standardizing  the  contrasts 
with  the  rank  nearest  the  0.683  percentile  as  his  initial  estimate  of  "a", 
he  used  the  slope  of  the  regression  line  calculated  with  the  smallest 
standardized  contrasts  up  to  and  including  the  contrast  at  the  rank  nearest  to 
the  0.683  percentile. 

In  his  Versions  XR  and  SR,  Zahn  wanted  to  see  if  replotting  and 
reassessing  would  improve  the  power  of  the  half-normal  plot.  In  Version  XR, 
if  the  largest  contrast  were  found  to  exceed  the  guardrail,  the  remaining 
ordered  contrasts  would  be  restandardized  and  replotted  using  as  an  estimate 
of  "a"  the  value  at  the  rank  nearest  the  0.683  percentile  for  the  remaining 
k-1  contrasts  and  replotting  on  a  new  scale  for  k-1  points.  This  process  is 
iterated  until  no  effects  are  found  to  exceed  the  guardrail.  In  Version  SR, 
at  each  stage  of  the  detection  process,  "c"  is  estimated  from  the  slope 
of  [0.67  k'J  against  k'  normal  order  statistical  values,  where  k’  is  the 
number  of  contrasts  being  examined  for  significance,  a  number  which  reduces  at 
each  iteration.  When  no  more  significant  effects  are  determined,  then  k*  ■  e. 

Zahn  noted  that  all  of  the  versions  are  based  on  the  assumption  that  the 
investigator  knows  how  many  contrasts  are  real  and  how  many  are  error  (p. 

203).  He  tried  another  version,  R,  in  which  he  also  revised  the  initial 
estimate  of  "a"  after  the  largest  effect  of  any  k’  effects  is  declared 
significant  and  before  the  next  ranked  contrast  is  tested.  In  Version  R,  Zahn 
estimated  "o"  from  a  slope  based  on  half  the  number  of  the  contrasts  being 
examined.  This  allowed  him,  therefore,  to  include  up  to  half  of  the  original 
k  contrasts,  rather  than  0.7  k,  as  in  Version  X. 

Others  have  investigated  different  techniques  for  estimating  o  in  an 
unreplicated  design.  These  include  the  efforts  by  Lloyd  (1952)  using 
generalized  least  squares  Instead  of  ordinary  least  squares,  and  by  Wilk, 


Gnanadesikan,  and  Freeny  (1963)  using  a  maximum  likelihood  estimate  of 
a2.  None  seemed  to  justify  their  application. 

ZAHN'S  INVESTIGATIONS.  Zahn  (1975b)  performed  an  empirical  Monte  Carlo  study 
—  1000  runs  —  of  half-normal  plots  limited  to  the  k  =  15  contrast  case.  He 
first  generated  a  set  of  pseudo-normal  standard  normal  deviates  to  represent 
the  15  error  contrasts.  He  then  compared  the  sensitivity  of  the  five 
techniques  —  versions  —  in  detecting  real  effects  at  different  a  settings 
when  the  number  and  size  of  the  real  contrasts  were  varied. 

Whereas  Daniel  (1959)  had  investigated  situations  in  which  only  one  real 
contrast  was  present,  ranging  in  size  from  one  "a"  to  six  "o",  Zahn 
studied  two  situations:  Type  1  Situation:  r  real  contrasts,  all  of  size  d; 

Type  2  Situation:  r  real  contrasts  all  of  size  d  and  r'  real  contrasts  all  of 
size  d'.  For  the  Type  1  study,  r  equalled  1,  2,  4,  and  6  real  contrasts  out 
of  15  contrasts  and  d  equalled  from  zero  to  8  "cr."  In  the  Type  II 
situation,  r  and  r'  both  equalled  either  one  or  two,  and  d  equalled  2,  4,  or  6 
"a"  and  d'  equalled  4,  6,  or  8  "o."  The  sensitivity  of  each  version  was 
tested  for  a  =  0.05,  0.20,  and  0.40. 

Zahn  also  compared  the  above  versions  with  the  nomination  approach.  In 
that  approach,  the  investigator  decides  a  priori  to  combine  the  higher  order 
interactions  to  form  an  estimate  of  error  variance  (Pearce,  1953).  As  stated 
in  the  introduction  to  this  paper,  the  assumption  must  be  made  that  these 
higher  order  effects  are  negligible  and  that  they  are  not  confounded  with 
lower-order  effects,  and  if  it  not  a  valid  assumption,  the  results  are  flawed. 

Zahn  used  two  criteria  for  evaluating  a  version's  performance:  (1) 
Detection  rate,  i.e.,  the  proportion  of  real  contrasts  present  in  a  situation 
which  are  detected,  and  (2)  False  positive  behavior,  the  probability  that  at 
least  one  null  contrast  is  declared  significant,  or  the  PER. 

ZAHN'S  CONCLUSIONS.  It  is  difficult  to  draw  Sharp  conclusions  from  Zahn's 
(  1975b)  results,  although  he  does  so  to  some  extent  within  the  limits  of  his 
investigation.  Still  it  is  apparent  from  the  data  that  there  are  interactions 


among  many  of  the  factors  being  manipulated.  Results  often  depend  on  what 
values  of  n,  r,  and  d  are  Involved. 

Some  of  his  conclusions  are: 

1.  Detection  rates  decrease  as  size  of  the  real  effects  contrasts 
decrease. 

2.  Detection  rates  decrease  as  the  number  of  real  contrasts  increases. 

3.  Size  and  number  of  real  contrasts  interact  in  their  effect  on  the 
detection  rates  (p.  206). 

4.  Increasing  a  from  0.05  to  0.40  increases  the  detection  rate  but  the 
probability  of  getting  false  positives  also  increases. 

5.  Though  detection  rate  varied  considerably  from  version  to  version, 
all  false  positive  rates  remain  close  to  their  respective  a's.  The 
largest  differences  occur  when  the  one  real  contrast  is  small  (p. 
204). 

6.  Version  R  tended  to  be  poorer  than  the  other  versions  when  the  number 
of  real  effects  were  small  (1  to  4  out  of  15  contrasts),  but  unlike 
the  other  versions,  it  did  not  totally  collapse  when  there  were  six 
real  effects.  However,  its  detection  rate  is  still  low  and  the 
estimate  of  "a"  is  biased  (p.  210).  This  could  be  explained  by  the 
fact  that  when  there  are  6  real  contrasts  out  of  only  15,  the  chances 
increase  that  the  denominator  of  the  test  statistic  would  include  the 
smaller  real  contrasts  (p.  205). 

7.  Versions  XR  and  SR  yielded  poorer  detection  rates  when  compared  with 
Versions  X  and  S  respectively  when  two  or  four  real  effects  were 
present  in  the  15  contrasts.  In  addition  to  being  poorer,  the 
iterative  versions  require  more  calculations  (p.  205).  Zahn 
concluded  that  revising  the  estimate  of  "o"  after  each  significant 
result  did  not  improve  the  detection  rate  and  is  not  recommended  (p. 


Among  all  versions,  Zahn  recommended  using  Version  S  (p.  205), 
although  when  an  inspection  is  made  of  his  tables,  it  can  be  seen 
that  in  some  cases  this  superiority  is  not  large  (particularly  when 
compared  with  Version  X),  and  in  some  cases,  it  may  not  exist  at  all. 

Zahn  recommended  using  the  half-normal  plot  rather  than  the 
nomination  approach  if  equivalent  PERsare  desired,  unless  almost  all 
null  contrasts  can  be  accurately  nominated  (p.  210).  For  screening 
purposes,  one  cannot  use  the  nomination  approach  because  the 
higher-order  interactions  are  confounded  with  lower-order  effects  so 
that  both  can  bias  one  another.  This  confounding  is  temporarily 
acceptable  in  screening  experiments,  since  with  a  sequential 
methodology,  one  expects  to  eventually  discover  and  isolate  these 
effects. 

Overall,  Zahn's  results  suggest  that  detection  rates  are  relatively 
poor  in  situations  of  considerable  pract  cal  interest,  that  is,  when 
there  are  more  than  just  a  few  real  effects  and  when  the  effects  were 
not  too  large.  In  Table  1,  partial  data  from  Zahn’s  Table  5  (1975b, 
p.  207)  shows  the  detection  rate,  PER,  and  final  estimate  of 
"o"  for  Version  S  with  two  or  four  real  effects  of  size  d  =  2"o", 
s-cf",  or  6mam  critical  values  of  0.05,  0.20,  and  0.40. 
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TABLE  1.  EMPIRICAL  BEHAVIOR  OF  VERSION  S  IN  2-  and  4-SITUATIONS  WITH  K 

REAL  CONTRASTS  OF  SIZES  2 "cr",  4"a",  and  6"<3"  PRESENT, 

USING  a  =  0.05,  0.20,  and  0.40 

(From  Zahn's  Table  5  [1975b,  p.  207])  / 

.n 

-  s 


Number  of 
Real 

Contrasts 

Present 

Criterion 

a 

Size  of  the  Real  Contrasts  Present 

2"o"  4*0"  6"o" 

i 

jiS 

Detection 

0.05 

.080 

.539 

.954 

Jl 

Rate 

0.20 

.219 

.821 

.997 

t 

0.40 

.365 

.920 

1.000 

2 

PER 

0.05 

.023 

.044 

.046 

0.20 

.121 

.166 

.181 

■i 

0.40 

.283 

.341 

.335 

£ 

Final  Estimate 

0.05 

1.148 

1.089 

1.004 

¥ 

of  "O" 

0.20 

1.117 

1.019 

.982 

0.40 

1.069 

.971 

.954 

J* 

Detection 

0.05 

.037 

.270 

.864 

( 

Rates 

0.20 

.134 

.654 

.988 

0.40 

.247 

.835 

.999 

4 

PER 

0.05 

.008 

.002 

.000 

,1 

0.20 

.050 

.010 

.000 

AJ 

0.40 

.143 

.028 

.000 

,*v 

Final  Estimate 

0.05 

1.335 

1.432 

1.124 

of  "a" 

0.20 

1.297 

1.235 

1.002 

r*  * 

0.40 

1.236 

1.111 

1.013 
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SECTION  III 


RATIONALE  FOR  THE  PRESENT  PROJECT 

Daniel  and  Zahn's  work  on  half-normal  plots  became  the  starting  point  for 
the  present  project.  There  appeared  to  be  a  number  of  reasons  to  examine 
further  the  operating  characteristics  of  both  normal  and  half-normal  plots  and 
to  evaluate  how  plots  might  be  employed  when  applied  to  the  unique  features 
that  characterize  the  screening  experiment. 

Specifically,  the  major  motivation  for  the  present  project  derived  from 
the  following  beliefs  and  for  the  reasons  given: 


■f. 


1.  Little  work  had  been  done  to  investigate  the  operating  characteristics  of 
half-normal  plots  since  Zahn's  work  in  1975. 

Zahn  (1977)  was  asked  if  he  were  aware  of  any  additional  investigations  of 
the  operating  characteristics  of  the  half-normal  following  his  work.  He 
answered  that  after  1975  he  did  not  continue  in  that  line  of  research  but  that 
he  was  not  aware  of  any  work  that  had  been  done. 

In  a  cursory  review  of  the  literature  in  which  Daniel’s  and  Zahn's  work 
are  referred  to  (see  Appendix  A),  it  is  interesting  to  note  that  out  of  the 
more  than  100  papers  listed  in  the  citation  indexes,  Zahn's  work  is  only 

referred  to  by  others  three  times.  In  reports  in  which  the  half-normal  plot 
was  used  to  test  significance,  decisions  were  generally  made  by  "eye-balling" 
the  data  rather  than  by  using  the  more  precise  guardrails. 


2.  Up  to  now,  most  of  the  work  on  plots  as  a  test  of  significance  had  been 
with  the  half-normal  plots,  while  the  full-normal  plot  had  been  used  to 
evaluate  data  quality. 


After  using  the  half-normal  plot  for  some  time,  Daniel  (1976)  decided  that 
a  great  many  defects  in  experimental  data  "are  strongly  sign  dependent,  and 
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all  are  properties  of  subsets  of  the  data  set  which  are  obscured  in  the 
half-normal  plots  by  overaggregation"  (p.  149).  Daniel  and  others  dropped  the 
half-normal  plot  and  began  to  use  the  full-normal  primarily  for  evaluating 
data  quality  rather  than  as  an  aid  to  the  detection  process. 

Draper  and  Smith  (1981)  described  both  the  normal  and  half-normal  plot  as 
a  means  of  evaluating  residuals  but  without  using  guardrails  to  facilitate 
decisions  regarding  outliers. 

3.  Zahn's  investigation  was  limited  primarily  to  smaller  designs  than  are 
likely  to  be  used  in  screening  experiments,  suggesting  that  certain 
limitations  noted  by  him  might  not  apply  with  the  larger  designs. 

The  bulk  of  Zahn's  (1975b)  empirical  study  of  the  half-normal  plot  had 
been  limited  to  the  2^  designs,  where  k  =  4,  and  n  *  16.  When  he  concludes 
that  the  plots  might  be  useful  If  the  number  of  real  eEfects  were  limited  to 
four,  it  was  because  four  contrasts  represented  one-fourth  of  his  data  and 
there  would  be  only  11  contrasts  with  which  to  estimate  the  error  variance. 
When  he  frequently  detected  less  than  60%  of  the  real  effects  which  were  two 
to  four  times  the  size  of  the  standard  error  of  the  contrasts  (which,  for  his 
work,  translates  into  one  and  two  times  the  size  oE  his  population  a),  this 
too  reflects  the  fact  that  with  the  smaller  design,  sensitivity  will  tend  to 
be  poorer. 

It  seemed  desirable  to  determine  whether  the  same  problems  would  occur 
proportionately  when  the  size  of  the  experiment  was  increased.  Thus,  while 
four  factors  may  decrease  the  sensitivity  of  a  design  with  15  contrasts,  eight 
are  unlikely  to  have  as  strong  an  effect  on  one  with  31  contrasts,  and  the 
chances  of  picking  out  suitable  null  contrasts  should  increase.  Furthermore, 
it  seemed  Important  to  consider  plot  effectiveness  when  dealing  more  with  real 
effects  of  marginal  size. 
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are  more  concerned  with  avoiding 


>e  II  errors  than  avoiding 


>e  I  during 


the  detection  process. 


In  a  screening  experiment,  the  investigator  is  more  concerned  about 
rejecting  potentially  critical  factors  than  he  is  about  erroneously  accepting 
some  null  ones.  At  this  phase  of  the  experimental  program,  his  data  is  not  so 
precise  that  he  should  dare  to  eliminate  factors  with  marginal  effects.  As 
Daniel  suggested,  for  screening  we  should  select  our  critical  values  nearer 
the  0.40  a  level  than  at  the  0.05  level.  A  reexamination  of  the  data  with 
an  emphasis  on  reducing  Type  II  errors  at  the  expense  of  making  Type  I  errors 
seemed  justified. 


5.  Plots  frequently  provide  the  data  in  a  form  that  is  easier  to  interpret 
than  the  conventional  F-test  and  in  some  cases  may  be  the  only  technique 


available. 


There  is  no  information  inherent  in  the  graphic  approach  that  cannot  be 
obtained  from  the  data  in  numerical  form.  However,  there  is  evidence  to 
suggest  that  broad  interpretations  of  the  data  can  often  be  made  more  easily 
using  the  plots;  on  the  other  hand,  precision  is  more  readily  obtained  using 
numerical  data.  The  visual  presentation  of  plotted  data,  properly  organized, 
can  enable  an  investigator  to  see  relationships  and  distortions  that  he  might 
otherwise  miss.  Ordering  by  magnitude  provides  another  overview  of  the  data. 
It  allows  the  investigator  to  quickly  perceive  groupings  of  factors  —  i.e., 
main  and  interaction  effects  —  that  have  the  greatest  effect  on  performance. 
Such  observations  tend  to  favor  the  use  of  the  plots  in  a  screening  study, 
where  "ease  of  calculation  is  often  more  important  than  slight  differences  in 
stringency  of  conclusion"  (Kurtz,  Link  et  al.,  1965).  One  big  advantage  of 
plots  is  that,  without  the  benefit  of  statistics,  they  automatically  adjust 
for  multiplicity,  which  the  (mis)user  of  the  F-test  may  overlook  when  he  tests 
a  great  many  effects,  one  at  a  time,  without  adjusting  for  the  fact  that  he 
would  probably  get  some  significant  effects  strictly  by  chance.  With  an 
unreplicated  design,  of  course,  no  estimate  of  the  error  variance  is  available 
for  doing  the  conventional  F-test. 


SECTION  IV 


CONSTRUCTING  NORMAL  AND  HALF-NORMAL  PLOTS 

Normal  and  half-normal  plots  are  graphic  methods  which  allow  the 
Investigator  to  visually  compare  an  empirically  derived  cumulative 
distribution  of  the  effects  with  a  cumulative  distribution  derived  from  a 
normally  (or  ha If -normally)  distributed  population.  Once  the  contrasts  have 
been  calculated,  the  plots  are  constructed  using  normal  probability  paper. 

One  difference  between  the  normal  and  the  half-normal  plot  lies  in  the 
fact  that  the  latter  contains  only  the  unsigned  values  of  the  k  contrasts  from 
a  2^  p  factorial  experiment  while  the  former  retains  the  signs.  Another 
difference  is  that  the  half-normal  plot  is  essentially  a  folded  normal  plot,  a 
fact  that  requires  the  plotting  positions  to  change  accordingly. 

CONSTRUCTING  NORMAL  PLOTS 

The  normal  plot  for  k  contrasts  from  a  2E  P  fractional  factorial 

[f-p] 

experiment  (where  k  »  n  -  1  ■  2  -l)is  constructed  in  the  following 

manner  on  normal  probability  paper: 

1.  The  k  contrasts  are  plotted  in  order  of  magnitude,  with  signs  intact, 
on  the  linear  scale  of  a  normal  probability  grid.  Whether  this  scale  is 
placed  on  the  ordinate  or  the  abscissa  is  somewhat  arbitrary.  Zahn  (1975a) 
suggested  that  the  contrasts  should  be  plotted  on  the  ordinate  in  order  to 
make  the  half-normal  grid  correspond  more  closely  to  the  usual  regression 
analysis  graph  on  which  the  random  variable  is  plotted  as  the  ordinate;  this 
same  argument  would  apply  to  the  normal  plot.  Since  investigators  using  these 
plots  to  present  their  experimental  results  have  done  it  both  ways,  a  reader 
should  be  careful  to  determine  which  is  being  used  in  order  not  to 
misinterpret  the  patterns  of  plotted  points.  In  one  case,  real  effects  fall 
off  below  the  zero  line  and  in  the  other,  above  the  zero  line.  The  author  saw 
one  unsophisticated  user  become  confused  and  misinterpret  his  own  data, 
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claiming  a  number  of  effects  to  be  real  when.  In  fact,  the  plots  fell  on  the 
wrong  side  of  the  line,  a  sign  of  truncated  data,  not  significant  contrasts. 

2.  The  points  are  plotted  against  the  percentiles  of  the  standard  normal 
distribution  on  the  other  axis.  The  particular  plotting  positions  of  the 
normal  distribution  can  be  obtained  from  a  table  of  expected  values  of  normal 
order  statistics  (Owen,  1962,  for  up  to  n  3  50;  Harter,  1961,  for  up  to  99). 

If  one  needs  expected  values  for  sets  larger  than  99,  tables  of  expected 
values  of  normal  order  statistics  may  be  generated  using  a  Monte  Carlo 
simulation.  The  expected  values  of  normal  and  half-normal  order  statistics 
are  given  for  n  *  32,  63,  and  127  in  Appendix  B.  The  programs  used  to 
generate  expected  values  of  order  statistics  for  this  report  are  given  in 
Appendix  C  (normal  plots)  and  Appendix  D  (half-normal  plots).  Barnett  (1976) 
discusses  several  methods  of  estimating  these  values,  including  an 
alternative,  simple  yet  fully  efficient,  estimation  method  proposed  by  Gupta 
(1952). 

An  example  of  a  normal  plot  is  shown  in  Figure  2.  Three  effects  —  two 
positive  and  one  negative  —  are  presumably  real  because  of  their  distance 
from  the  null  line. 

P(%)  i 


Signed  Contrasts 

Figure  2.  Example  of  Normal  Plot 
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CONSTRUCTING  HALF-NORMAL  PLOTS 

In  the  half-normal  plot,  the  absolute  contrasts  —  values  with  the  signs 
Ignored  —  are  plotted  against  the  percentiles  of  the  standard  half-normal 
distribution. 

Percentiles  for  plotting  positions  of  the  half-normal  distribution  may  be 
obtained  from  tables  (Zahn,  1975a,  between  n  =  6  and  20),  simple  calculations 
(Draper  and  Smith,  1966),  or  a  computer  program  (Sparks,  1970;  with  a 
modification  by  Munford,  1972).  The  computer  program  used  for  this  report  is 
given  in  Appendix  D.  Leone,  Nelson,  and  Nottingham  (1961)  also  discuss  some 
methods  for  estimating  the  expected  values  for  the  "folded  normal 
distribution."  Equation  1  may  also  be  used  to  generate  the  expected  values 
for  the  half-normal  distribution,  as  Daniel  did. 
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Figure  3.  Example  of  Half-Normal  Plot 


(Using  same  data  as  in  Figure  2) 


The  data  used  to  generate  the  normal  plot  In  Figure  2  was  modified  to 
generate  an  example  of  the  half-normal  plot  shown  in  Figure  3. 

STANDARDIZED  PLOTS 

Instead  of  the  raw  contrasts,  the  investigator  may  wish  to  plot  the 
standardized  contrasts  on  which  the  guardrails,  proposed  by  Daniel,  may  also 
be  plotted.  In  theory,  these  can  be  obtained  by  satisfying  the  following 
equation: 

Obtained  contrast 

Standardized  contrast  =* 

a  V  4/n  (6) 


where  o  is  the  population  standard  deviation  and  n  is  the  number  of 
conditions  in  the  experiment  from  which  the  contrasts  were  generated.  In 
practice,  however,  the  population  standard  deviation  is  not  known  and  an 
estimated  value,  s,  is  used  instead.  With  unreplicated  designs,  as  the  review 
of  Daniel's  and  Zahn’s  work  revealed,  it  is  not  an  easy  nor  obvious  task  to 
find  an  accurate  s. 


4.  Calculate  the  k  »  (n  -  1)  contrasts  from  the  data  in  Step  3  using 
Yates*  algorithm. 

The  above  four  steps  produce  a  set  of  randomly  distributed  contrasts  — 
the  mean  differences  between  the  two  conditions  of  each  experimental  effect  in 
the  2C  design  —  from  a  null  experiment.  The  standard  error  of  the 
contrasts  equals: 

s  =  a  sTVZ  (7) 

c 

where  a  is  the  population  a,  which  is  1.00  in  this  simulation,  and  n  is 
the  number  of  independent  data  points  in  the  experiment.  In  the  unreplicated 
experiment,  n  equals  the  number  of  independent  experimental  conditions.  In 
the  replicated  experiment,  n  equals  the  total  number  of  conditions.  The 
derivation  of  Equation  7  is  given  in  Appendix  E. 

To  introduce  real  effects  into  the  data: 

5.  Create  r  real  effects  which  may  all  be  of  the  same  or  different 
magnitudes  (d^),  of  which  r+  are  positive  and  r-  are  negative. 

6.  The  r  effects  are  added  to  the  signed  contrasts  which  were  randomly 
assigned  (in  Step  3)  to  the  first  r  conditions  of  the  design  arranged 
in  Yates’  standard  order.  This  assignment  procedure  for  real  effects 
is  acceptable  since  no  identification  of  specific  effects  is  required 

7.  The  final  contrasts,  the  composites  of  error  and  real  effects,  are 
ordered  by  the  magnitude  of  the  signed  contrasts  (when  studying  the 
normal-order  plot)  and  of  the  unsigned  contrasts  (when  studying  the 
half-normal  plots. 

In  the  Monte  Carlo  program,  the  above  steps  were  repeated  5000  times,  the 
equivalent  of  running  an  experiment  5000  times.  The  mean  and  standard 
deviation  of  the  contrasts  occurring  at  each  rank  position  of  the  ordered 
contrasts  were  obtained.  Also,  the  real  effects  were  tracked  and  the 


number  of  times  a  real  effect  (regardless  of  magnitude)  fell  at  each  rank 
position  was  totalled. 

BOUNDARIES  OF  THE  EXPERIMENTAL  SPACE  EXAMINED  IN  THIS  PROJECT 

Since  much  of  this  effort  was  exploratory,  a  broad,  multidimensional 
experimental  space  was  examined  unevenly,  sampling  relevant  portions  as 
questions  arose. 

Factor  parameters  included: 

Size  of  experimental  design:  n  =  32,  64,  128 

Number  of  ordered  contrasts:  k  =  31,  63,  127 

Number  of  real  effects  r  =  0,  1,  2,  3,  4,  5,  6,  7,  8,  12,  16 

Number  of  positive  effects  r+  =  The  number  ranged  from  100%  r  down  to 

50%  r  in  unit  steps. 

Size  of  effects:  d  =  0.50O,  1.00O,  1.150  1.25o, 

1.330,  1.50O,  1.670, 

2.000  and  occasionally 
intermediate  values. 

EXAMPLE 

An  example  of  the  data  from  the  above  procedure  for  a  normal  plot  is  given 
in  Table  2  for  the  situation: 

n  *  Design  size  =  Number  of  experimental  conditions  =  2^  *  32 
k  =  Number  of  effects  (contrasts  or  mean  differences)  =  2^  -  1  *  31 
r  =  Number  of  real  effects  =  16  [Number  of  error  contrasts,  e  =  15] 
r+  =  Number  of  positive  real  effects  =  16 
d  =  Magnitude  of  real  effects  =  1.00 


TABLE  2.  EXAMPLE  OF  ANALYSES:  CONTRAST,  NUMBER  OF  REAL  EFFECTS, 
STANDARDIZED  CONTRAST,  AND  CONTRAST  STANDARD  DEVIATION  AT  EACH  RANK 


SITUATION:  n= 

NORMAL 

32,  k  = 

PLOT  DATA 

31,  r=16+0-,  d 

=  1.00 

STANDARDIZED 

STD. 

# 

CONTRAST 

#  R 

CONTRAST 

DEVIATION 

1 

-*0.612 

1 

-1.730 

0.194 

2 

-0.441 

4 

-1.246 

0.148 

3 

-0.335 

5 

-0.947 

0.131 

4 

-0.252 

16 

-0.714 

0.123 

5 

-0.180 

34 

-0. 510 

0.117 

6 

-0.117 

43 

-0.331 

0.  115 

7 

-0.058 

75 

-0.164 

0.114 

8 

-0.003 

111 

-0.009 

0.112 

9 

0.054 

169 

0.154 

0.112 

10 

0.  112 

239 

0.317 

0.112 

11 

0.172 

393 

0.486 

0.112 

12 

0.234 

564 

0.661 

0.  112 

13 

0.300 

876 

0.849 

0.113 

14 

0.373 

1334 

1.054 

0.115 

15 

0.452 

2029 

1.278 

0.115 

16 

0.533 

2958 

1.507 

0.116 

17 

0.611 

3692 

1.730 

0.115 

18 

0.683 

4092 

1.931 

0.114 

19 

0.  748 

4438 

2.115 

0.111 

20 

0.809 

4641 

2.288 

0.110 

21 

0.865 

4789 

2.446 

0.108 

22 

0.  921 

4816 

2.606 

0.109 

23 

0.973 

4883 

2. 753 

0.107 

24 

1.027 

4936 

2.  905 

0.109 

25 

1.081 

4  946 

3.056 

0.110 

26 

1.137 

4961 

3.  216 

0.113 

2  7 

1.198 

4979 

3.388 

0.116 

2  8 

1  .  265 

4985 

3.577 

0.  121 

* 

1.346 

4995 

3.  806 

0.130 

.  .  4  c  1 

4  996 

4.103 

0.146 

1  . 

1  .  6  2  2 

5  0  0  0 

4.588 

0.191 

36 


Three  sets  of  data  were  obtained:  (1)  The  mean  contrasts  at  each  position 

of  the  ordered  data,  x.  .  ;  (2)  The  standard  deviation  of  those  means  at  each 

X  #  K 

position,  s  ;  and  (3)  The  total  number  of  real  contrasts  out  of  5000  runs 

X » K 

that  appear  at  each  rank  of  the  ordered  data,  #R  . 

X  #  IC 

The  percentage  of  real  effects  that  might  occur  at  any  rank  In  this 
simulation  Is: 

%R  =  [#R  +  5000]  X  100.  (8) 

i,k  i,k 

We  shall  refer  to  the  real  contrasts  that  fall  at  a  rank  where  only  error 
contrasts  were  intended  as  "R-spillover. "  We  shall  refer  to  the  error 
contrasts  that  fall  at  a  rank  where  only  the  real  effects  were  intended  as 
E-spillover.  The  E-spillover  is  obtained  for  the  appropriate  ranks  by 
subtracting  the  %R  from  100%  or  the  #R  from  5000. 

X  r  iC  1  #  K 

Standardized  values  of  the  mean  contrasts,  z.  ,  were  also  derived  from 

X  0  K 

the  raw  contrasts  using  the  following  equation: 


where 


z  =  expected  standardized  contrast  for  rank  i  of  k  cases 
i  f  k 

x  *  the  expected  contrast  for  rank  i  of  k  cases 

1  r  IC 

a  *  population  a 

n  =  number  of  experimental  conditions  in  the  design 


In  this  phase  of  the  analysis,  where  results  are  aggregates  of  the  results 
from  5000  runs,  the  correct  population  sigma,  i.e.,  a  =  1.00,  was  used  in 
Equation  9  as  in  Equation  7.  In  practice,  and  in  the  single  experiment,  it  is 
unlikely  that  this  value  would  be  known  and  so  would  have  to  be  estimated,  s. 
We  will  also  work  with  knowledge  of  how  many  real  effects  there  are  and  at 
what  ranks  of  the  ordered  contrasts  they  were  intended  to  be  located  before 
being  combined  with  the  error  component. 
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ACCURACY 


While  5000  runs  for  the  Monte  Carlo  studies  seemed  like  a  lot  initially  — 
Zahn  had  used  only  1000  runs  in  his  study  —  minor  variations  were  observed 
when  the  same  situations  were  recomputed  or  when  computations  were  compared 
with  other  tables  of  expected  values  of  normal  order  statistics  or  when  the 
theoretical  symmetry  was  not  exact  when  expected  at  opposite  jends  of  ordered 
data  internal  to  certain  tables.  These  variations  were  reduced  when  the  Monte 
Carlo  studies  were  made  with  10,000  runs. 

Careful  study  assured  us  that  if  the  slight  inaccuracy  of  the  5000  runs 
were  accepted,  there  would  be  no  alteration  of  any  conclusion  drawn  in  this 
report.  By  accepting  the  smaller  number  of  runs,  of  course,  computer  costs 
and  computing  times  were  reduced  considerably. 

As  a  rule-of-thumb  regarding  the  accuracy  of  our  data,  most  of  the  time, 
contrasts  and  standardized  contrasts  were  accurate  to  the  second  decimal  place 
when  rounded.  As  for  the  percent  of  real  contrasts  at  each  rank  of  the 
ordered  contrasts,  the  numbers  for  the  larger  values  seldom  varied  more  than 
one  or  occasionally  2%  from  one  set  of  data  to  another  or  within  runs  at  each 
end  of  ordered  contrasts  which  were  theoretically  the  same.  This  latter 
inaccuracy  is  most  visible  in  summary  tables  in  which  R-spillover  values  are 
given;  this  is  because  one  set  of  data  was  used  to  prepare  the  tables  while  a 
second  one,  calculated  with  identical  parameters,  was  used  to  provide  clean 
copy  for  publication. 

RELATING  ZAHN'S  PARAMETERS  TO  THOSE  USED  IN  THE  PRESENT  INVESTIGATION 

To  relate  certain  conditions  drawn  by  Zahn  to  those  reached  in  the  present 

investigation,  it  is  necessary  to  use  a  common  scale  when  referring  to  the 

size  of  an  effect.  When  Zahn  referred  to  a  real  contrast  "of  size  2o,"  he 

meant  that  the  effect  was  two  times  the  size  of  the  standard  error  of  the 

contrasts  (o  )  in  his  experiment  where  n  =  16.  When  we  refer  to  a  real 
c 

contrast  in  this  project  "of  size  2o,"  we  mean  that  the  effect  is  two  times 
the  size  of  the  population  a  which  is  independent  of  the  size  of  the 


simulated  data  collection  effort.  This  occurred  because  Zahn  developed  his  15 
contrasts  directly  by  sampling  from  a  standard  normal  population  of  N[0,1], 
while  in  this  project,  the  16  "scores*  for  the  null  experiment  were  obtained 
from  a  standard  normal  population  of  N[0,1]  and  from  those,  the  15  contrasts 
were  calculated.  Zahn's  population  cr,  to  which  he  did  not  refer,  was 
actually  equal  to  2.00,  while  ours  was  1.00. 

If  one  were  only  discussing  the  data  within  a  study,  which  scale  is  used 
would  be  less  important.  If,  however,  we  wish  to  relate  the  results  from  the 
two  studies,  the  scales  must  be  standardized. 

Actually,  when  we  express  the  size  of  a  real  effect  in  these  simulations, 
three  variables  are  involved,  of  which  the  third  is  determined  by  the  other 
two.  These  variables  are  the  population  sigma,  the  size  of  the  experiment, 
and  the  effect  size.  But  effect  size  must  be  expressed  in  terms  of  population 
sigma  or  standard  error  of  the  contrast.  Since  the  latter  is  a  function  of 
population  sigma  and  experiment  size,  when  contrast  size  is  scaled  by  it,  we 
cannot  independently  study  the  effects  of  real  contrast  size  and  experiment 
size.  Because  we  ordinarily  have  no  control  over  the  population  sigma  in 
practice,  it  does  us  little  good  to  be  able  to  study  independently  the  effect 
of  real  contrast  size  and  changes  in  the  population  sigma. 

For  those  reasons,  we  will  scale  the  measures  from  both  studies  in  terms 
of  the  population  a,  since  that  is  more  in  line  with  what  would  happen  in 
the  real  world  and  because  that  will  allow  us  to  isolate  the  effects  of  real 
contrast  size,  d,  and  the  experiment  size,  n. 

One  can  relate  the  sizes  of  Zahn's  effects  to  ours  by  converting  them  both 
to  a  multiple  of  population  o.  Since  he  only  used  experiments  of  size  n  * 

16,  to  express  the  size  of  his  real  effects  in  terms  of  the  a,  as  used  in 
this  project,  it  is  only  necessary  to  divide  his  values,  which  are  in  terms  of 
sc  by  two.  In  those  terms,  the  effect  of  the  population  sigma  remains 
constant,  while  the  effects  of  contrast  size  and  experiment  size  can  be 
isolated. 
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SECTION  VI 


ANALYSES  AND  RESULTS 


Simulated  data  for  2  experimental  designs  were  created  and  analyzed  In 
order  to: 


1.  Examine  characteristics  of  expected  values  of  normal  order  plots  as  a 
function  of  experiment  size,  n;  number  of  ordered  contrasts,  k;  number  of  real 

effects,  r;  size  of  real  effects,  d  ;  and  the  number  of  positive  real 

r 

effects,  r+. 

2.  Compare  the  relative  effectiveness  of  normal  and  half-normal  plots. 

3.  Examine  individual  plots. 

4.  Provide  guardrails  for  evaluating  half-normal  plots  for  experiments 
of  size,  n  «  32  and  64. 

CHARACTERISTICS  OF  NORMAL  PLOT  DATA 

The  first  analysis  was  carried  out  in  order  to  examine  the  characteristics 
of  expected  values  of  ordered  contrasts  as  a  function  of  the  size  of  the 
experiment,  n;  the  number  of  ordered  contrasts,  k,  which  equal  (n  -  1)  in  an 
unreplicated  design;  the  number  of  real  effects,  r;  the  number  of  positive 
real  effects,  r+;  and  the  size(s)  of  the  real  effects  (dr). 

Let  us  not  be  too  concerned  with  quantification  and  pay  more  attention  to 
the  patterns  formed  by  the  data.  For  starters,  let  us  look  at  the  mean 
contrast  data  for  these  parameters: 

n  =  32,  k  =  31,  r  =  8,  d  =  +2a 


S'. 


•v' 

*<** 


£ 


j- 

w* 


£ 


fi 


41 


The  ordered  mean  contrasts,  the  number  of  times  a  real  effect  appears  at 
each  position  (out  of  5000  runs),  the  contrast  standard  deviations  of  the  new’ 
contrasts  at  each  rank,  and  the  standardized  mean  contrasts  are  given  in  Table 
3-a. 

For  this  aggregate  data  from  5000  cases,  the  eight  real  contrasts  are 
easily  distinguishable  from  the  error  contrasts.  The  spillover  of  real 
effects  into  the  ranks  where  only  error  contrasts  are  intended  were  trivial  in 
this  example.  Only  0.3%  of  the  real  effects  were  located  at  the  23rd  (error) 
rank  out  of  5000  runs. 

Overall,  the  largest  contrasts  are  the  most  variable.  The  contrast 
standard  deviations  at  each  rank  show  that  variability  is  greatest  at  both 
ends  of  the  intended  error  contrasts  (ranks  1  and  23),  being  nearly  double  of 
what  is  found  at  the  center  of  that  set  of  23  contrasts  (rank  12). 
Unfortunately,  these  contrast  standard  deviations  are  not  from  normal 
distributions,  particularly  those  of  the  larger  and  more  important  contrasts. 
Therefore,  we  cannot  use  them  to  establish  the  probability  that  a  contrast  at 
any  particular  rank  would  fall  a  certain  distance  from  the  expected  value. 
Efforts  to  employ  these  contrast  standard  deviations  to  estimate  the 
R-spillover  at  particular  ranks  proved  too  complex  and  inaccurate  to  pursue 
(see  Appendix  F). 

Examination  of  the  contrasts  reveals  an  almost  symmetrical  set  of  values 
between  ranks  1  and  23,  with  rank  12  at  the  pivot  point.  Since  ranks  1 
through  23  were  intended  to  be  the  error  contrasts  in  our  simulation,  and 
since  there  is  practically  no  spillover  of  real  effects  into  these  ranks  in 
this  example,  it  is  not  surprising  to  find  that  these  standardized  "error" 
contrasts  are  almost  identical  —  within  the  two  decimal  point  accuracy  of  our 
data  —  to  the  expected  values  for  normal  order  statistics  (e.v.n.o.s.)  for  n 
=  23.  These  values  are  given  in  Table  4-a. 
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TABLE  3.  EXAMPLE  OF  ORIGINAL  AND  STANDARDIZED  ORDERED 
CONTRASTS,  #Rik,  AND  CONTRAST  STANDARD  DEVIATIONS 


r.v* 


TABLE  4.  EXPECTED  VALUES  OF  NORMAL  ORDER  STATISTICS  FOR  n  *  23  and  n  -  1 19. 


ORDER 
STATISTIC 
-1.92916 
-1.48137 
-1.21445 
-1.01356 
-0.84697 
-0.70115 
-0.56896 
-0.44609 
-0.32965 
-0.21755 
-0.10813 
0.00000 
0. 10813 
0.21755 
0.32965 
0.44609 
0.56896 
0.70115 
0.84697 
1.01356 
1.21445 
1.48137 
1.92916 


ORDER 

|  STATISTIC 

1  -2.56913 

2  -2.21696 

3  -2.02022 

4  -1.87972 

5  -1.76869 

6  -1.67595 

7  -1.59577 

8  -1.52463 

9  -1.46051 

10  -1.40183 

11  -1.34775 

12  -1.29728 

13  -1.24987 

14  -1.20522 

15  -1.16277 

16  -1.12237 

17  -1.08368 

18  -1.04661 

19  -1.01088 

20  -0.97638 

21  -0.94304 

22  -0.91077 

23  -0.87936 

24  -0.84880 

25  -0.81900 

26  -0.78996 

27  -0.76152 

28  -0.73371 

29  -0.70645 

30  -0.67970 

31  -0.65344 

32  -0.62763 

33  -0.60223 

34  -0.57720 

35  -0.55252 

36  -0.52820 

37  -0.50418 

38  -0.48042 

39  -0.45697 

40  -0.43374 

41  -0.41075 

42  -0.38800 

43  -0.36541 

44  -0.34301 

45  -0.32079 

46  -0.29874 

47  -0.27682 

48  -0.25504 

49  -0.23338 

50  -0.21183 

51  -0.19037 

52  -0.16900 

53  -0.14771 

54  -0.12649 

55  -0.10532 

56  -0.08420 


-0.06312 

-0.04206 

-0.02103 


0.02103 
0.04206 
0.06312 
0.08420 
0. 10532 
0.12649 
0. 14771 
0.16900 
0.19037 
0.21183 
0. 23339 
0.25505 
0.27683 
0.29874 
0.32080 
0.34301 
0.36541 
0.38800 
0.41075 
0.43375 
0.45696 
0.48042 
0.50418 
0.52821 
0.55253 
0.57720 
0.60223 
0.62763 
0. 65344 
0.67971 
0.70645 
0.73371 
0.76153 
0.78997 
0.81900 
0.84881 
0.87935 
0.91078 
0.94306 
0.97639 
1. 01088 
1.04662 
1.08367 
1.12238 
1.16277 
1.20523 
1  .  24988 
1.29727 
1.34777 
1.40184 
1.46049 
1.52465 
1.59576 
1.67593 
1 . 76871 
1.87974 
2.02022 
2.21696 
2.56913 
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What  happens  when  some  of  the  contrasts  are  positive  and  some  are 
negative?  In  Table  3-b,  mean  contrasts  were  generated  for  the  following 
situation: 

n  *  32,  k  *  31,  r  ■  8,  d  *  +2.00 o,  d  *  -2.00o 

•  2 

Once  again  it  is  apparent  that  with  real  effects  of  this  size  and  number,  the 
R-spillover  is  trivial.  This  time,  however,  some  spillover  occurs  at  both 
ends  of  the  scale. 

EFFECT  OF  MARGINAL  SIZE  EFFECTS.  If  plots  are  to  be  used  for  screening 
studies,  they  must  be  helpful  in  detecting  real  effects  that  are  marginal  in 
size.  Let  us  look  at  what  happens  to  the  estimated  contrasts  and  the 
R-spillover  when  the  sizes  of  the  effects  decrease.  To  do  this,  we  will  hold 
the  other  parameters  fixed  for  the  moment  and  use  all  positive  real  effects. 
The  contrasts  in  Table  5  are  average  values  over  5000  runs  for  this  series  of 
situations: 

n  =  32,  k  =  31,  r  =*  8,  and  one  table  each  for  d  =  +0.50o 
+1.00O;  +1.05a;  +1.15a;  +1.33a;  +1.50O;  and  +1.67d. 

As  the  sizes  of  the  real  effects  decrease,  the  probability  increases  that 
some  real  contrasts  will  be  located  at  ranks  other  than  the  first  eight 
positive  ones  where  they  were  purposely  placed  for  this  simulation.  Among  the 
intended  error  contrasts,  the  more  R-spillover  at  any  rank,  the  further  the 
expected  contrast  (mean  of  5000  runs)  at  that  rank  deviates  from  the 
e.v.n.o.s.  for  the  number  of  error  contrasts,  i.e.,  23. 


TABLE  5.  CHANGES  IN  RESULTS  AS  A  PUNCTION  OF  THE 
MAGNITUDE  OF  REAL  EFFECTS 


(a) 


(b) 


NORMAL  PLOT  DATA 


NORMAL  PLOT  DATA 


SITUATION:  n=  32,  k  =  31,  r=  8+0-,  d=0.50 


SITUATION:  n«  32,  k«  31,  r-  8+0-,  d«1.00 


STANDARDIZED 

STD. 

1 

CONTRAST 

*  R 

CONTRAST 

DEVIATION 

1 

-0.683 

44 

-1.931 

0.179 

2 

-0.527 

87 

-1.491 

0.135 

3 

-0.433 

103 

-1.225 

0.118 

4 

-0.362 

136 

-1.024 

0.108 

S 

-0.305 

180 

-0.862 

0.101 

6 

-0.255 

214 

-0. 722 

0.098 

1 

-0.209 

270 

-0.593 

0.093 

8 

-0. 167 

301 

-0.472 

0.091 

9 

-0.127 

345 

-0.361 

0.088 

10 

-0.091 

370 

-0.257 

0.087 

11 

-0.056 

443 

-0.158 

0.086 

12 

-0.022 

513 

-0.061 

0.085 

13 

0.012 

580 

0.035 

0.084 

14 

0.046 

643 

0.130 

0.084 

IS 

0.080 

730 

0.225 

0.084 

16 

0.113 

802 

0.319 

0.084 

17 

0.147 

959 

0.415 

0.084 

18 

0.182 

1106 

0.  514 

0.085 

19 

0.218 

1101 

0.616 

0.085 

20 

0.254 

1287 

0.719 

0.086 

21 

0.292 

1517 

0.825 

0.087 

22 

0.  331 

1680 

0.936 

0.089 

23 

0.  372 

1817 

1.053 

0.091 

24 

0.416 

2029 

1.175 

0.094 

25 

0.463 

2276 

1.309 

0.098 

26 

0.516 

2481 

1.458 

0.  102 

27 

0.576 

2850 

1.630 

0.110 

28 

0.645 

3213 

1.824 

0.118 

29 

0.727 

3573 

2.057 

0.132 

30 

0.839 

3981 

2.  374 

0.  154 

31 

1.023 

4369 

2.894 

0.206 

STANDARDIZED 

STD. 

1 

CONTRAST 

#  R 

CONTRAST 

DEVIATION 

1 

-0.687 

0 

-1.944 

0.184 

2 

-0.526 

1 

-1.488 

0.139 

3 

-0.432 

3 

-1.222 

0.122 

4 

-0.361 

5 

-1.020 

0.112 

5 

-0.302 

5 

-0.854 

0.105 

6 

-0.250 

7 

-0.708 

0.101 

7 

-0.203 

8 

-0.573 

0.097 

8 

-0.159 

8 

-0.450 

0.095 

9 

-0.118 

10 

-0.332 

0.094 

10 

-0.079 

19 

-0.222 

0.093 

11 

-0.040 

18 

-0.113 

0.092 

12 

-0.001 

32 

-0.004 

0.092 

13 

0.036 

50 

0.103 

0.091 

14 

0.074 

81 

0.210 

0.091 

15 

0.114 

81 

0.321 

0.092 

16 

0.153 

116 

0.434 

0.093 

17 

0.  196 

127 

0.554 

0.094 

18 

0.241 

198 

0.680 

0.097 

19 

0.289 

276 

0.816 

0.099 

20 

0.343 

423 

0.969 

0. 102 

21 

0.403 

654 

1.139 

0.107 

22 

0.474 

1034 

1.340 

0.112 

23 

0.556 

1662 

1.573 

0.119 

24 

0.654 

2853 

1.851 

0.  128 

25 

0.758 

3686 

2.144 

0.139 

26 

0.859 

4335 

2.430 

0.141 

27 

0.959 

4630 

2.713 

0.144 

28 

1.060 

4800 

2.997 

0.148 

29 

1.171 

4918 

3.313 

0.157 

30 

1.305 

4974 

3.690 

0.174 

31 

1.503 

4986 

4.250 

0.215 

TABLE  5.  (cont’d) 


(c) 


w 


normal  plot  data 

SITUATION:  n*  32,  k-  31,  r-  8+0-,  d*1.0S 


|  CONTRAST 

.  n  C  Oft 


STANDARDIZED  STD. 

1  A  CONTRAST  DEVIATION 
l  -1.923  0.180 


NORMAL  PLOT  DATA 

SITUATION:  n-  32,  k«  31,  r«  8+0-,  d-1.15 


|  CONTRAST  #  R 
1  -0.684  « 

i  _a  c-J-5  1 


STANDARDIZED 
CONTRAST 
-1.934 
-1. 


STD. 

DEVIATION 

0.181 

0.137 


TABLE  5-  (cont’d) 


(e) 


(f) 


SITUATION:  n= 

NORMAL 

32,  k- 

PLOT  DATA 

31,  r-  8+0-, 

d-1.25 

STANDARDIZED 

STD. 

1 

CONTRAST 

t  R 

CONTRAST 

DEVIATION 

I 

-0.685 

0 

-1.938 

0.183 

2 

-0. S26 

0 

-1.488 

0.138 

3 

-0.429 

0 

-1.214 

0.120 

4 

-0.358 

0 

-1.012 

0.110 

5 

-0.299 

0 

-0.847 

0.104 

6 

-0.248 

0 

-0.702 

0.099 

7 

-0.201 

0 

-0.568 

0.096 

8 

-0.158 

0 

-0.446 

0.095 

9 

-0.117 

0 

-0.331 

0.093 

10 

-0.078 

2 

-0.221 

0.092 

11 

-0.040 

1 

-0.113 

0.092 

12 

-0.001 

7 

-0.003 

0.092 

13 

0.  037 

10 

0.106 

0.092 

14 

0.076 

14 

0.214 

0.093 

15 

0.114 

14 

0.323 

0.094 

16 

0.156 

10 

0.442 

0.095 

17 

0.200 

26 

0.566 

0.097 

18 

0.  246 

41 

0.697 

0. 100 

19 

0.296 

55 

0.837 

0.104 

20 

0.  3S3 

126 

0.998 

0.108 

21 

0.421 

215 

1.191 

0.116 

22 

0.507 

464 

1.433 

0.125 

23 

0.625 

1141 

1.767 

0.142 

24 

0.815 

3629 

2.304 

0.162 

25 

0.963 

4537 

2.724 

0.158 

26 

1.085 

4815 

3.069 

0.152 

27 

1.196 

4929 

3.382 

0.153 

28 

1.304 

4974 

3.687 

0.151 

29 

1.417 

4995 

4.007 

0.157 

30 

1.550 

4996 

4.384 

0.171 

SITUATION:  n- 

NORMAL 

32,  k- 

PLOT  DATA 

31,  r»  8+0-, 

d*l. 33 

STANDARDIZED 

STD. 

1 

CONTRAST 

1  R 

CONTRAST 

DEVIATION 

1 

-0.682 

0 

-1.930 

0.181 

2 

-0.525 

0 

-1.486 

0.136 

3 

-0. 432 

0 

-1.222 

0.120 

4 

-0.361 

0 

-1.022 

0.  Ill 

5 

-0.303 

0 

-0.857 

0.105 

6 

-0.252 

0 

-0.713 

0.101 

7 

-0.203 

2  - 

-0.576 

0.098 

8 

-0.159 

0 

-0.450 

0.096 

9 

-0.117 

1 

-0.332 

0.093 

10 

-0.078 

1 

-0.222 

0.092 

11 

-0.038 

3 

-0.108 

0.091 

12 

0.001 

1 

0.003 

0.091 

13 

0.039 

3 

0.111 

0.092 

14 

0.078 

2 

0.220 

0.093 

15 

0.117 

9 

0.331 

0.094 

16 

0.  158 

6 

0.447 

0.094 

17 

0.201 

7 

0.568 

0.097 

18 

0.247 

12 

0.700 

0.099 

19 

0.299 

54 

0.845 

0.102 

20 

0.359 

62 

1.015 

0.  108 

21 

0.426 

140 

1.206 

0.115 

22 

0.  514 

285 

1.454 

0.128 

23 

0.641 

950 

1.812 

0.149 

24 

0.876 

3939 

2.478 

0. 173 

25 

1.042 

4694 

2.946 

0.165 

26 

1.172 

4889 

3.316 

0.156 

27 

1.282 

4960 

3.626 

0.153 

28 

1.391 

4984 

3.935 

0.153 

29 

1.501 

4999 

4.244 

0.160 

30 

1.633 

4998 

4.619 

0.174 

TABLE  5.  (cont’d) 


(g) 


(h) 


NORMAL  PLOT  DATA 


NORMAL  PLOT  DATA 


SITUATION:  n=*  32,  k  =  31,  r-  8+0-,  d=1.50 


SITUATION:  n»  32,  k=  31,  r»  8+0-, 


#  CONTRAST  |  r 

1  -0.681  0 

2  -0.522  0 

3  -0.429  0 

4  -0.358  0 

5  -0.298  0 

6  -0.247  0 

7  -0.201  0 

8  -0.157  0 

9  -0.117  1 

10  -0.077  0 

11  -0.038  0 

12  0.000  0 

13  0.038  0 

14  0.077  0 

15  0.116  0 

16  0.158  1 

17  0.202  5 

18  0.249  6 

19  0.300  7 

20  0.359  16 

21  0.429  47 

22  0.520  97 

23  0.664  460 

24  1.021  4498 

25  1.205  4896 

26  1.335  4975 

27  1.448  4993 

28  1.554  4998 

29  1.666  5000 

30  1.802  5000 

31  2.004  5000 


STANDARDIZED 

STD. 

CONTRAST 

DEVIATION 

-1.926 

0.185 

-1.478 

0.136 

-1.214 

0.119 

-1.013 

0.  110 

-0.843 

0.104 

-0.699 

0.099 

-0.568 

0.097 

-0.444 

0.095 

-0.330 

0.093 

-0.217 

0.093 

-0.108 

0.092 

0.000 

0.092 

0.108 

0.092 

0.217 

0.092 

0.328 

0.093 

0.447 

0.094 

0.571 

0.095 

0.704 

0.099 

0.848 

0.103 

1.017 

0.109 

1.214 

0.118 

1.470 

0.134 

1.879 

0.163 

2.887 

0.  192 

3.408 

0.168 

3.775 

0.157 

4.096 

0.152 

4.396 

0.152 

4.712 

0.159 

5.098 

0.  177 

5.669 

0.219 

1 

CONTRAST 

1  R 

1 

-0.683 

0 

2 

-0.527 

0 

3 

-0.429 

0 

4 

-0.358 

0 

5 

-0.300 

0 

6 

-0.249 

0 

7 

-0.202 

0 

8 

-0.159 

0 

9 

-0.118 

0 

10 

-0.079 

0 

11 

-0.039 

0 

12 

-0.001 

0 

13 

0.038 

0 

14 

0.076 

0 

15 

0.116 

0 

16 

0.157 

1 

17 

0.201 

0 

18 

0.247 

2 

19 

0.298 

2 

20 

0.357 

2 

21 

0.428 

8 

22 

0.522 

40 

23 

0.675 

217 

24 

1.174 

4754 

25 

1.368 

4981 

26 

1.502 

4994 

27 

1.616 

4999 

28 

1.725 

5000 

29 

1.837 

5000 

30 

1.976 

5000 

31 

2.176 

5000 

STANDARDIZED 

CONTRAST 

-1.931 

-1.490 

-1.215 

-1.014 

-0.849 

-0.704 

-0.572 

-0.450 

-0.334 

-0.224 

-0.111 

-0.002 

0.107 

0.216 

0.327 

0.443 

0.567 

0.697 

0.843 

1.010 

1.211 

1.476 

1.910 

3.322 

3.868 

4.249 

4.571 

4.878 

5.197 

5.588 

6.156 


=  1.67 


STD. 

DEVIATION 
0.182 
0.137 
0.119 
0.111 
0.105 
0.  101 
0.097 
0.094 
0.092 
0.091 
0.091 
0.091 
0.091 
0.092 
0.093 
0.095 
0.097 
0.100 
0.105 
0.111 
0.120 
0.138 
0.177 
0.  206 
0.171 
0.155 
0.152 
0.152 
0.159 
0. 176 
0.218 


This  can  be  seen  in  an  examination  of  the  23  error  contrasts  in  Table  5-b 


(in  which  the  eight  real  effects  are  all  of  size,  d  ■  lo).  They  are  no 
longer  symmetrical  about  rank  12.  If  we  compare  them  to  the  23  e.v.n.o.s.  in 
Table  4,  we  see  that  the  contrasts  closest  in  rank  to  the  real  effects,  rank 
23,  is  approximately  0.80  of  the  expected  value  and  at  rank  22,  0.90. 

With  the  smaller  real  effects,  the  degrading  of  the  expected  contrasts 
from  the  correct  number  of  e.v.n.o.s.  is  more  apparent.  At  ranks  23,  22,  and 
21  in  Table  5-b,  the  spillover  of  real  effects  into  those  intended-to-be  error 
ranks  was  appr  ximately  33%,  21%,  and  14%  respectively.  That  last  value  means 
that  in  one  of  seven  experiments  with  31  contrasts  and  eight  real  effects  of 
size  =  +1.00s,  the  eleventh  largest  effect  will  be  real.  Conversely, 

7%  of  the  time,  the  fifth  largest  ranking  contrast  will  not  be  real. 

A  summary  table.  Table  6,  provides  an  overview  of  the  data  in  Tables  5-a 
through  5-h. 


TABLE  6.  SPILLOVER  OF  REAL  AND  ERROR  CONTRASTS 
AS  A  FUNCTION  OF  EFFECT  SIZE 
(Situation  n  =  32,  k  *  31,  r  *  8,  all  positive) 


Effect 

Effect 

Rank  Where  R-Sp 11 lover 

Percent  E-Spillover 

Size 

Size 

Exceeds 

this 

Percent 

at 

Rank 

■~Vn 

(d  sc) 

(d  a) 

5% 

10% 

15% 

20% 

25% 

#24 

#25 

#26 

#27 

4.72 

1.67 

X 

X 

X 

X 

X 

4 

X 

X 

X 

4.24 

1.50 

23 

X 

X 

X 

X 

11 

X 

X 

X 

o 

3.76 

1.33 

22 

23 

X 

X 

X 

21 

6 

2 

1 

\'v 

3.53 

1.25 

22 

23 

23 

23 

X 

28 

10 

3 

1 

3.25 

1.15 

21 

22 

23 

23 

23 

34 

15 

6 

3 

*  , 

2.97 

1.05 

20 

21 

22 

23 

23 

40 

22 

11 

5 

2.82 

1.00 

19 

21 

22 

22 

23 

44 

26 

13 

7 

1.41 

0.50 

7 

12 

15 

18 

20 

59 

54 

50 

43 

Table  6  shows,  for  example,  that  among  31  contrasts,  of  which  eight  are 

real,  If  all  real  effects  are  of  size  d  =  1.15o  (or  3.25s  ),  we  might 

c 

expect  to  find  one  of  them,  not  among  the  eight  largest  contrasts  of  the 
ordered  values,  but  ranking  tenth  in  size  more  than  10%  of  the  time. 

Conversely,  15%  of  the  time,  the  seventh  largest  contrast  would  actually  be  an 
error  contrast.  Since  the  numbers  in  Table  6  are  all  dependent  on  the  size  of 
the  experiment  and  the  number  and  sizes  of  the  real  effects,  their  value  to 
the  reader  is  only  to  show  that  caution  is  required  when  viewing  a  set  of 
ordered  contrasts  and  assuming  the  largest  are  the  real  effects,  even  when 
they  deviate  from  the  null  line. 

EFFECT  OF  HAVING  MIXED  POSITIVE  AND  NEGATIVE  EFFECTS.  Let  us  increase  the 
realism  of  our  simulated  data  by  introducing  some  negative  real  effects  as 
well.  We  have  seen  what  happens  when  all  real  effects  are  bunched  at  the 
positive  end.  The  data  in  Tables  7-a  through  7-d  allows  us  to  see  how  the 
contrasts  and  the  spillover  of  real  and  error  contrasts  into  unintended 
locations  change  as  the  proportion  of  negative  effects  change.  For  this 
table,  the  following  situations  were  generated. 

n  =  32,  k  =  31,  r  =  8,  d  =  1.000,  with  1,  2,  3.  or  4  of  the  eight 

real  effects  being  negative. 

Scanning  the  four  tables  that  make  up  Table  7,  one  can  see  that  more 
E-spillover  and  R-spillover  occur  at  that  end  of  the  ordered  contrasts  where 
more  real  effects  are  intended  to  be  located.  For  example,  in  Table  7-a,  at 
the  negative  end  where  only  one  real  negative  effect  was  added,  11%  of  the 
runs  would  have  real  contrasts  located  at  the  first  adjacent  error  rank,  #2. 

At  the  other  end,  where  seven  positive  real  contrasts  were  added,  32%  of  the 
runs  would  have  real  contrasts  located  at  the  first  adjacent  error  rank,  #24. 

Furthermore,  the  23  mean  contrasts  in  the  ranks  intended  to  contain  error 
contrasts  are  no  longer  symmetrical  as  the  expected  values  in  Table  4  are. 
Instead,  they  are  asymmetrical  when  the  number  of  positive  and  negative  real 
effects  are  uneven  and  are  more  degraded  from  their  expected  values  on  the  end 
where  the  most  real  effects  were  originally  added. 


NORMAL  PLOT  DATA 

SITUATION:  n=  32,  k=  31,  r=  7+1-,  d=1.00 


STANDARDIZED  STD. 

|  CONTRAST  #  R  CONTRAST  DEVIATION 

1  -1.044  3905  -2.953  0.290 

2  -0.657  S52  -1.858  0.165 

3  -0.517  225  -1.462  0.132 

4  -0.426  130  -1.205  0.119 

5  -0.356  64  -1.007  0.111 

6  -0.299  34  -0.847  0...10S 

7  -0.249  39  -0.705  0.100 

8  -0.202  21  -0.571  0.097 

9  -0.158  31  -0.447  0.094 

10  -0.117  12  -0.330  0.093 

11  -0.077  20  -0.219  0.092 

12  -0.039  29  -0.109  0.091 

13  0.000  35  -0.001  0.090 

14  0.038  42  0.107  0.091 

15  0.077  50  0.217  0.091 

16  0.115  74  0.326  0.091 

17  0.156  103  0.441  0.093 

18  0.198  127  0.559  0.094 

19  0.243  182  0.688  0.097 

20  0.292  265  0.827  0.100 

21  0.345  408  0.976  0.104 

22  0.404  603  1,143  0.107 

23  0.474  892  1.340  0.114 

24  0.561  1610  1.588  0.124 

25  0.670  2873  1.895  0.136 

26  0.783  3777  2.215  0.145 

27  0.896  4411  2.534  0.150 

28  1.008  4708  2.852  0.153 

29  1.125  4858  3.181  0.162 

30  1.266  4942  3.581  0.173 

31  1.479  4978  4.182  0.216 


NORMAL  PLOT  DATA 

SITUATION:  n=»  32,  k  =  31,  r=  6+2-,  d*=1.00 


1 

CONTRAST 

#  R 

STANDARDIZED 

CONTRAST 

STD. 

DEVIATION 

1 

-1.203 

4682 

-3.403 

0.277 

2 

-0.874 

3445 

-2.471 

0.  218 

3 

-0.627 

869 

-1.775 

0.154 

4 

-0.506 

391 

-1.432 

0. 13U 

5 

-0.421 

215 

-1.191 

0.116 

6 

-0.353 

126 

-0.999 

0.108 

7 

-0.295 

98 

-0.835 

0.103 

8 

-0.244 

59 

-0.691 

0.099 

9 

-0.199 

34 

-0.562 

0.096 

1  0 

-0.155 

31 

-0.439 

0.094 

11 

-0.115 

30 

-0.326 

0.093 

12 

-0.076 

29 

-0.216 

0.092 

13 

-0.038 

25 

-0.107 

0.092 

14 

0.001 

37 

0.002 

0.092 

15 

0.039 

35 

0.110 

0.092 

16 

0.078 

50 

0.220 

0.092 

17 

0.117 

67 

0.330 

0.095 

18 

0.157 

79 

0.443 

0.096 

19 

0.200 

113 

0.565 

0.098 

20 

0.245 

193 

0.693 

0. 100 

21 

0.294 

247 

0.831 

0.102 

22 

0.348 

363 

0.984 

0. 106 

23 

0.411 

545 

1.163 

0.111 

24 

0.484 

840 

1.369 

0. 117 

25 

0.575 

1475 

1.627 

0.128 

26 

0.693 

2885 

1.960 

0. 143 

27 

0.821 

3916 

2.  323 

0.155 

28 

0.948 

4469 

2.683 

0.159 

29 

1.081 

4757 

3.057 

0.168 

30 

1.  232 

4916 

3.483 

0.184 

31 

1.450 

4979 

4.102 

0.227 

*0  V’  w"  -  ft  m  <r  _  •  *  „  ■»  * 


TABLE  7  (cont’d) 


NORMAL  PLOT  DATA 

SITUATION:  n=  32,  k=  31,  r=  5+3-,  d»1.00 


,  NORMAL  PLOT  DATA 

SITUATION:  n»  32,  k=*  31,  r-  4+4-,  d=1.00 


I  CONTRAST 

1  -1.300 

2  -1.017 

3  -0.797 

4  -0.614 

5  -0.499 

6  -0.418 

7  -0.352 

8  -0.296 

9  -0.246 

10  -0.200 
LI  -0.156 
L  2  -0.116 

L3  -0.076 
L  4  -0.038 

15  0.000 

16  0.038 

L  7  0.076 

18  0.115 

19  0.156 

20  0.199 

21  0.245 

22  0.294 

23  0.348 

24  0.413 

25  0.490 

26  0.586 

27  0.719 

28  0.863 

29  1.018 

30  1.183 

31  1.414 


STANDARDIZED 
CONTRAST 
-3.676 
-2.876 
-2.254 
-1.735 
-1.411 
-1.182 
-0.997 
-0.838 
-0.695 
-0.565 
-0.442 
-0.327 
-0.21S 
-0.107 
0.001 
0.  108 
0.216 
0.326 
0.442 
0.562 
0.692 
0.833 
0.984 
1.168 
1.386 
1.657 
2.033 
2.442 
2.878 
3.345 
4.000 


STD. 

DEVIATION 
0.258 
0.217 
0.183 
0.143 
0.125 
0. 114 
0.106 
0.102 
0.099 
0.096 
0.094 
0.093 
0.092 
0.091 
0.091 
0.091 
0.092 
0.092 
0.094 
0.096 
0.098 
0. 101 
0.105 
0.112 
0.121 
0. 133 
0.153 
0.166 
0.179 
0.194 
0.238 


t  CONTRAST 

1  -1.370 

2  -1.118 

3  -0.929 

4  -0.755 

5  -0.602 

6  -0.496 

7  -0.416 

8  -0.350 

9  -0.295 

10  -0.244 

11  -0.199 

12  -0.157 

13  -0.116 

14  -0.077 

15  -0.039 

16  -0.001 

17  0.038 

18  0.075 

19  0.116 

20  0.156 

21  0.199 

22  0.245 

23  0.296 

24  0.350 

25  0.416 

26  0.495 

27  0.596 

28  0.754 

29  0.924 

30  1.110 

31  1.365 


STANDARDIZED 
CONTRAST 
-3.876 
-3.163 
-2.627 
-2.136 
-1.702 
-1.403 
-1.178 
-0.991 
-0.834 
-0.691 
-0.563 
-0.443 
-0.329 
-0.218 
-0.110 
-0.002 
0.106 
0.214 
0.328 
0.441 
0.562 
0.694 
0.836 
0.990 
1.175 
1.401 
1.687 
2.132 
2.613 
3.  139 
3.861 


STD. 

DEVIATION 
0.244 
0.203 
0.184 
0.165 
0.139 
0. 121 
0.111) 

0. 104 
0.099 
0.095 
0.094 
0.093 
0.092 
0.092 
0.091 
0.091 
0.092 
0.092 
0.0  93 
0.094 
0.097 
0.099 
0.101 
0.105 
0.113 
0.  123 
0.137 
0. 166 
0.187 
0.205 
0.24U 


In  a  real  experiment  the  data  will  neither  be  as  smooth  as  this  aggregate 
data  nor  will  we  know  how  many  effects  are  real  nor  their  sign.  However,  we 
can  expect  fewer  spillovers  on  the  end  where  the  fewer  obvious  effects  are 
visible. 

EFFECT  "'F  MIXING  THE  SIZES  OF  THE  REAL  EFFECT.  Up  to  this  point,  in  order  to 
simplify  the  discussion,  we  have  used  simulations  in  which  all  real  effects 
were  of  the  same  size.  In  practice  this  is  unlikely  to  occur.  Therefore,  the 
information  in  these  tables  are  illustrative  at  best  and  of  limited  practical 
value. 

Let  us  see  how  the  contrasts  and  spillover  patterns  change  when  the  real 
effects  are  of  different  magnitudes.  For  this,  let  us  examine  this  situation: 

n  =  32,  k  =  31,  r  =  8,d  (one  for  each  r)  =  +1.000,  +l,00o,  +1.33o, 

+1.330,  +1.670,  +1.670,  +2.00 O,  +2.000. 

The  results  are  shown  in  Table  8. 

It  would  be  a  valuable  tool  if  a  limited  number  of  stylized  tables  of  the 
type  shown  in  this  report  could  be  prepared  from  which  an  investigator  might 
draw  data  which  he  could  then  combine  to  fit  his  particular  situation.  Could 
we  have  predicted  these  results  from  the  tables  that  we’ve  already  generated? 
To  what  extent  does  the  data  in  Table  8  (where  the  average  of  the  eight 
effects  sizes  equals  1.5)  resemble  the  data  in  Table  5-g  where  d^  *»  +1.50? 

To  what  extent  does  the  data  in  Table  8  resemble  the  averages  of  values  from 
four  sets  of  tables  (Table  5-b,  5-f,  5-h,  and  3-a)  where  all  eight  of  the 
real  effects  are  of  size  1.000,  1.33o,  1.67c,  and  2.000,  respectively? 

Fewer  real  effects  spilled  into  the  lower  ranks  when  all  eight  effects 
were  of  size  1.5o  (Table  5-g)  than  when  they  were  mixed,  but  with  sizes 
averaging  1.50  (Table  8).  The  smaller  size  real  effects  exert  a  much 
stronger  effect  on  the  data  than  the  larger  ones  in  the  mixed  data. 


TABLE  8.  RESULTS  WHERE  THE  SIZES  OF  THE  REAL  EFFECTS  ARE  MIXED 


SITUATION:  n= 

NORMAL 

32,  k  = 

PLOT  DATA 

31,  r=MI XED, 

d=l. 00 

STANDARDIZED 

STD. 

f 

CONTRAST 

#  R 

CONTRAST 

DEVIATION 

1 

-0.681 

0 

-1.927 

0.181 

2 

-0.525 

1 

-1.484 

0.  140 

3 

-0. 430 

0 

-1.216 

0.122 

4 

-0.359 

1 

-1. 016 

0.111 

5 

-0.300 

1 

-0.849 

0.106 

6 

-0.248 

2 

-0.702 

0.  102 

7 

-0.201 

0 

-0.569 

0.098 

8 

-0.157 

2 

-0.444 

0.096 

9 

-0.116 

2 

-0.327 

0.095 

10 

-0.076 

6 

-0.216 

0.093 

11 

-0.037 

4 

-0.105 

0.093 

12 

0.  000 

10 

0.000 

0.091 

13 

0.  039 

16 

0.109 

0.092 

14 

0.076 

16 

0. 216 

0.092 

15 

0.116 

29 

0.328 

0.093 

16 

0.158 

21 

0.446 

0.096 

17 

0.200 

42 

0.566 

0.098 

18 

0.246 

46 

0.696 

0.100 

19 

0.298 

91 

0.842 

0.104 

20 

0.354 

153 

1.000 

0.108 

21 

0.421 

246 

1.190 

0.115 

22 

0.  505 

433 

1.427 

0.127 

23 

0.625 

1042 

1.767 

0.149 

24 

0.827 

3398 

2.339 

0.186 

25 

1.037 

4554 

2.  934 

0.194 

26 

1.235 

4900 

3.492 

0.  186 

27 

1.411 

4985 

3.992 

0.182 

28 

1.589 

4999 

4  .  495 

0.183 

29 

1.775 

5000 

5.  020 

0.191 

30 

1.979 

5000 

5.598 

0.206 

31 

2.256 

5000 

6.380 

0.254 

r -  v -  v  *■  w -  '.'i  .tt, 


Nor  does  the  average  of  results  from  four  sets  of  data  (Tables  5-f,  5-h, 
and  3-a)  that  bracket  the  mixed  data  in  Table  8  adequately  represent  the  * 
results  from  the  mixed  data.  Efforts  to  improve  this  resemblance  using  log 
and  z-score  transformations  were  also  inadequate.  Here  again  the  smaller 
effects  have  a  stronger  influence  on  the  degree  of  spillover  and  the 
degradation  in  contrast  estimates.  In  fact,  the  mixed  data  fell  between  data 
where  all  effects  were  of  size  lo  and  1.33o,  illustrating  again  the 
greater  impact  of  the  marginal  effects. 

Although  not  totally  unexpected,  these  results  indicate  that  no  simple 
relationship  exists.  The  idea  of  having  a  limited  set  of  tables  from  which  an 
investigator  can  select  and  recombine  information  to  better  understand  his  own 
particular  results  is  not  likely  to  become  possible  without  considerable  more 
work,  if  at  all. 

EFFECT  OF  A  DIFFERENT  NUMBER  OF  REAL  EFFECTS.  Zahn  (1975b)  found  that  as  the 
number  of  real  effects  increased,  there  was  a  drop  in  the  detection  rate.  Let 
us  examine  the  effect  of  changing  r  while  holding  the  other  parameters 
constant,  for  the  following  situations: 


n  -  32,  k  -  31,  d  «  +1.250,  r  =  4;  5;  6;  7;  9;  10;  12;  or  16 
r 


The  mean  results  from  5000  runs  are  given  in  Table  9. 


As  the  number  of  real  effects  increase,  the  greater  the  chances  are  that 
one  or  more  of  the  larger  contrasts  are  not  real. 

EPFECT  OF  INCREASING  THE  SIZE  OF  THE  EXPERIMENT.  Since  increasing  n  in  an 
experiment  decreases  the  size  of  the  error  variance,  the  sensitivity  of  the 
test  for  significance  increases  and  more  real  effects  are  detectable.  The 
same  principle  should  apply  to  this  plot  data. 


The  following  situations  are  shown  in  Table  10. 


r  =  8,  d  =  +1.00o,  n  =  32,  64,  or  128,  and  k  =  (n  -  1). 


TABLE  9.  CHANGES  IN  RESULTS  AS  A  FUNCTION  OF  THE 
NUMBER  OF  REAL  EFFECTS 


I 

*"4 

r. 

i 


[ 


(a) 


(b) 


57 


r* - 


NORMAL 

PLOT  DATA 

NORMAL 

PLOT  DATA 

m 

SITUATION:  n= 

32,  k» 

31,  r«  4+0-,  d 

-1.25 

SITUATION:  n- 

32,  k- 

31,  r-  5+0-, 

d-1.25 

STANDARDIZED 

STD. 

STANDARDIZED 

STD. 

Mrl 

( 

CONTRAST 

t  R 

CONTRAST 

DEVIATION 

* 

CONTRAST 

#  .R 

CONTRAST 

DEVIATION 

1 

-0.711 

0 

-2.011 

0.179 

1 

-0.699 

0 

-1.977 

0.179 

2 

-0.552 

0 

-1.561 

0. 136 

2 

-0.545 

0 

-1.543 

0.133 

3 

-0.461 

0 

-1.305 

0.117 

3 

-0.454 

0 

-1.285 

0.117 

v*vj 

4 

-0.394 

0 

-1.114 

0. 108 

4 

-0.383 

0 

-1.083 

0.106 

5 

-0.338 

1 

-0.956 

0.101 

5 

-0.327 

0 

-0.926 

0.101 

^Y>-j 

6 

-0. 289 

0 

-0.817 

0.096 

6 

-0.278 

0 

-0.786 

0.095 

7 

-0.245 

0 

-0.693 

0.093 

7 

-0.233 

1 

-0.659 

0.091 

S 

-0.205 

0 

-0.581 

0.090 

8 

-0.193 

0 

-0.545 

0.089 

.\ 

9 

-0.169 

0 

-0.477 

0.088 

9 

-0.154 

0 

-0.435 

0.088 

f  *  ■'  .* 

10 

-0.134 

0 

-0.378 

0.087 

10 

-0.118 

0 

-0.334 

0.087 

S  | 

•  -  I 

11 

-0.099 

0 

-0.279 

0.086 

11 

-0.083 

1 

-0.235 

0.086 

/.-> 

12 

-0.065 

0 

-0.184 

0.085 

12 

-0.049 

2 

-0.138 

0.085 

13 

-0.033 

2 

-0.093 

0.085 

13 

-0.015 

4 

-0.042 

0.085 

14 

0.000 

1 

0.001 

0.084 

14 

0.021 

2 

0.058 

0.084 

WRsa 

15 

0.033 

3 

0.094 

0.085 

15 

0.053 

6 

0.151 

0.085 

16 

0.067 

4 

0.189 

0.085 

16 

0.088 

6 

0.248 

0.085 

/• 

17 

0.100 

5 

0.283 

0.085 

17 

0.122 

4 

0.345 

0.086 

.* 

18 

0.  135 

7 

0.381 

0.086 

18 

0.159 

11 

0.449 

0.088 

19 

0.171 

14 

0.483 

0.087 

19 

0.198 

12 

0.559 

0.090 

20 

0.209 

23 

0.591 

0.089 

20 

0.238 

17 

0.674 

0.092 

■  »* 

21 

0.249 

17 

0.705 

0.092 

21 

0.282 

38 

0.797 

0.096 

*  *  * 

m 

22 

0.292 

31 

0.825 

0.095 

22 

0.330 

55 

0.933 

0.099 

w 

23 

0.339 

42 

0.958 

0.099 

23 

0.384 

78 

1.087 

0.104 

24 

0.394 

66 

1.114 

0.104 

24 

0.  450 

140 

1.274 

0. 114 

25 

0.461 

126 

1.304 

0.113 

2  5 

0.535 

320 

1.513 

0.125 

1  -v'-’. 

26 

0.545 

284 

1.542 

0. 125 

26 

0.659 

915 

1.864 

0.147 

-V 

27 

0.671 

765 

1.899 

0.150 

27 

0.892 

3836 

2.524 

0.187 

' 

28 

0.938 

3894 

2.653 

0.198 

28 

1.080 

4661 

3.053 

0. 186 

29 

1.158 

4779 

3.274 

0.202 

29 

1.245 

4913 

3.521 

0.184 

■  ^ 

30 

1.359 

4945 

3.845 

0.210 

30 

1.421 

4979 

4.018 

0.192 

W  w- 

31 

1.614 

4991 

4.564 

0.247 

31 

1.654 

4999 

4.678 

0.233 

-  > ' 

**  ■ 

■  -  *  ' 

* 

,*-v 


$ 

$ 

am 


& 


_.*•  y*y*^  \« 


TABLE  9.  (cont'd) 


STANDARDIZED 

STD. 

t 

CONTRAST 

t  R 

CONTRAST 

DEVIATION 

1 

-0.694 

0 

-1.964 

0.179 

2 

-0.537 

0 

-1.520 

0.134 

3 

-0.445 

0 

-1.260 

0.117 

4 

-0.376 

1 

-1.062 

0.  108 

5 

-0.317 

0 

-0.898 

0.101 

6 

-0.267 

1 

-0.755 

0.096 

7 

-0.221 

1 

-0.626 

0.094 

8 

-0.181 

0 

-0.511 

0.092 

9 

-0.142 

1 

-0.402 

0.091 

10 

-0.104 

0 

-0.295 

0.089 

11 

-0.069 

5 

-0.194 

0.088 

12 

-0. 034 

1 

-0.095 

0.087 

13 

0.002 

3 

0.005 

0.087 

14 

0.036 

8 

0.102 

0.087 

15 

0.071 

3 

0.202 

0.088 

16 

0.  107 

3 

0.303 

0.088 

17 

0.144 

16 

0.408 

0.090 

18 

0.183 

14 

0.  518 

0.091 

19 

0.224 

27 

0.633 

0.094 

20 

0.269 

26 

0.762 

0.096 

21 

0.318 

49 

0.901 

0.100 

22 

0.375 

73 

1.062 

0. 104 

23 

0.443 

183 

1.252 

0.112 

24 

0.528 

386 

1.493 

0.124 

25 

0.647 

972 

1.830 

0.142 

26 

0.859 

3769 

2.428 

0.173 

27 

1.030 

4621 

2.912 

0.173 

28 

1.180 

4881 

3.337 

0.172 

29 

1.321 

4967 

3.735 

0.173 

30 

1.475 

4992 

4.  171 

0.  184 

31 

1.698 

4997 

4.802 

0.229 

STANDARDIZED 

STD. 

* 

CONTRAST 

1  R 

CONTRAST 

DEVIATION 

1 

-0.685 

0 

-1.936 

0.180 

2 

-0.531 

0 

-1.503 

0.133 

3 

-0.437 

0 

-1.236 

0.116 

4 

-0.368 

0 

-1.040 

0.107 

5 

-0.310 

0 

-0.876 

0.101 

6 

-0.258 

0 

-0.730 

0.098 

7 

-0.212 

0 

-0.599 

0.095 

8 

-0.170 

0 

-0.480 

0.093 

9 

-0.129 

0 

-0.365 

0.092 

10 

-0.091 

3 

-0.257 

0.091 

11 

-0.053 

2 

-0.149 

0.091 

12 

-0.015 

4 

-0.043 

0.090 

13 

0.021 

3 

0.060 

0.091 

14 

0.058 

4 

0.165 

0.091 

15 

0.096 

6 

0.272 

0.091 

16 

0.135 

12 

0.383 

0.092 

17 

0.175 

8 

0.495 

0.094 

18 

0.218 

21 

0.617 

0.095 

19 

0.263 

39 

0.744 

0.097 

20 

0.312 

81 

0.883 

0.100 

21 

0.369 

114 

1.043 

0.105 

22 

0.435 

226 

1.230 

0.113 

23 

0.517 

396 

1.463 

0.124 

24 

0.635 

1113 

1.796 

0.143 

25 

0.833 

3648 

2.356 

0.166 

26 

0.993 

4556 

2.808 

0.165 

27 

1.127 

4854 

3.188 

0.162 

28 

1.250 

4935 

3.535 

0.163 

29 

1.373 

4984 

3.883 

0.168 

30 

1.516 

4993 

4.289 

0.181 

31 

1.723 

4998 

4.875 

0.222 

w  *« 
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TABLE  9.  (cont’d) 


(e) 


(f) 


g 

■eu  rf  -  .f  -  r  -  -  - 


s  m  .• 


59 


*-£*< 
«.  i 


f ' 

NORMAL 

PLOT  DATA 

NORMAL' 

PLOT  DATA 

B 

K 

SITUATION:  n=* 

32,  k« 

31,  r»  9+0-,  d 

•1.25 

SITUATION:  n- 

32,  k»  31,  r-10+0-,  d 

•1.25 

; 

STANDARDIZED 

STD. 

STANDARDIZED 

STD. 

*  ■ 

» 

CONTRAST 

1  R 

CONTRAST 

DEVIATION 

# 

CONTRAST 

1  R 

CONTRAST 

DEVIATION 

•  /• 

1 

-0.675 

0 

-1.908 

0.183 

1 

-0.669 

0 

-1.891 

0.183 

■:•:• 

2 

-0.516 

0 

-1.459 

0.138 

2 

-0.509 

0 

-1.439 

0.139 

A 

V  _ 

L*  *4 

3 

-0. 419 

0 

-1.186 

0.121 

3 

-0.409 

0 

-1.157 

0.122 

4 

-0.348 

0 

-0.983 

0.111 

4 

-0.338 

0 

-0.956 

0.113 

■ 

5 

-0.287 

0 

-0.813 

0.104 

5 

-0.278 

0 

-0.785 

0.109 

& 

6 

-0.235 

0 

-0.664 

0.  100 

6 

-0.223 

0 

-0.632 

0.104 

gj 

• 

7 

-0.187 

0 

-0.529 

0.098 

7 

-0.174 

1 

-0.493 

0.101 

rr 

8 

-0. 143 

0 

-0.404 

0.097 

8 

-0.128 

1 

-0.363 

0.099 

► 

9 

-0.101 

1 

-0.287 

0.095 

9 

-0.084 

1 

-0.238 

0.097 

10 

-0.060 

2 

-0.171 

0.093 

10 

-0.041 

5 

-0.115 

0.096 

.;y 

^  . ' 

11 

-0.020 

6 

-0.058 

0.093 

11 

0.001 

6 

0.002 

0.096 

12 

0.020 

8 

0.057 

0.  093 

12 

0.043 

6 

0.121 

0.096 

*  y 

13 

0.060 

6 

0.169 

0.092 

13 

0.085 

6 

0.240 

0.097 

y  j 

14 

0.100 

13 

0.283 

0.093 

14 

0.129 

14 

0.365 

0.099 

■ 

15 

0.143 

16 

0.403 

0.09S 

IS 

0.174 

34 

0.492 

0.100 

h  I 

16 

0.187 

33 

0.528 

0.097 

16 

0.222 

57 

0.627 

0.102 

■y 

17 

0.235 

37 

0.664 

0.100 

17 

0.275 

69 

0.777 

0.106 

18 

0.286 

66 

0.810 

0.  103 

18 

0.335 

111 

0.947 

0.110 

19 

0.344 

113 

0.974 

0.109 

19 

0.404 

244 

1.142 

0.118 

*  -* 

20 

0.415 

246 

1.172 

0.117 

20 

0.491 

510 

1.389 

0.128 

21 

0.500 

494 

1.416 

0.128 

21 

0.608 

1278 

1.720 

0.141 

t 

22 

0.620 

1175 

1.753 

0.143 

22 

0.778 

3560 

2.202 

0.152 

23 

0.798 

3643 

2.258 

0.155 

23 

0.914 

4465 

2.585 

0.147 

_  , 

24 

0.941 

4506 

2.661 

0.150 

24 

1.026 

4795 

2.901 

0.  139 

*  y 

V> 

25 

1.052 

4796 

2.976 

0. 145 

25 

1.118 

4926 

3.162 

0.137 

*. 

26 

1.154 

4910 

3.265 

0.142 

26 

1.205 

4953 

3.407 

0.  136 

27 

1.252 

4959 

3.541 

0.141 

27 

1.291 

4975 

3.652 

0.136 

28 

1.350 

4978 

3.818 

0.  145 

28 

1.381 

4992 

3.905 

0.  139 

|v' 

29 

1.456 

4994 

4.119 

0.151 

29 

1.481 

4995 

4.188 

0.147 

a  *  ( 

£ 

* 

30 

1.584 

4998 

4.480 

0.  169 

30 

1.603 

4996 

4.534 

0.  163 

31 

1.781 

5000 

5.038 

0.214 

31 

1.794 

5000 

5.074 

0.206 
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STANDARDIZED 

STD. 

1 

CONTRAST 

f  R 

CONTRAST 

DEVIATION 

1 

-0.653 

0 

-1.846 

0.188 

2 

-0.489 

0 

-1.383 

0.142 

3 

-0.388 

1 

-1.099 

0.125 

4 

-0.312 

0 

-0.884 

0.117 

5 

-0.250 

1 

-0.708 

0.110 

6 

-0.193 

2 

-0.546 

0.  105 

7 

-0.142 

0 

-0.401 

0.103 

8 

-0.093 

0 

-0.264 

0.  101 

9 

-0.047 

7 

-0.132 

0.100 

10 

0.000 

7 

0.000 

0.099 

11 

0.046 

13 

0.131 

0.100 

12 

0.093 

14 

0.264 

0.100 

13 

0.143 

31 

0.403 

0.101 

14 

0.  193 

49 

0.545 

0.  104 

15 

0.248 

86 

0.701 

0.107 

16 

0.309 

119 

0.875 

0.111 

17 

0.380 

233 

1.076 

0.117 

18 

0.467 

518 

1.322 

0.  127 

19 

0.582 

1352 

1.645 

0.140 

20 

0.748 

3525 

2.  115 

0.149 

21 

0.877 

4487 

2.479 

0.145 

22 

0.979 

4768 

2.768 

0.137 

23 

1.066 

4888 

3.  016 

0.129 

24 

1.  144 

4953 

3.235 

0.127 

25 

1.217 

4964 

3.442 

0.125 

26 

1.287 

4986 

3.640 

0.  126 

27 

1.360 

4997 

3.848 

0.129 

28 

1.441 

5000 

4.075 

0.132 

29 

1.531 

4999 

4.331 

0.142 

30 

1.649 

5000 

4.663 

0.158 

31 

1.827 

5000 

5.168 

0.201 

STANDARDIZED 

STD. 

1 

CONTRAST 

»  R 

CONTRAST 

DEVIATION 

1 

-0.613 

0 

-1.733 

0.191 

2 

-0.442 

0 

-1.249 

0.150 

3 

-0.334 

0 

-0.946 

0.132 

4 

-0.252 

0 

-0.711 

0. 124 

5 

-0.183 

2 

-0.517 

0.118 

6 

-0.118 

7 

-0.333 

0.114 

7 

-0.058 

10 

-0.165 

0.111 

8 

-0.002 

18 

-0.005 

0.109 

9 

0.057 

26 

0.161 

0.110 

10 

0.117 

37 

0.330 

0.112 

11 

0.181 

81 

0.511 

0.115 

12 

0.252 

157 

0.711 

0.120 

13 

0.329 

279 

0.930 

0.126 

14 

0.422 

584 

1.195 

0.134 

15 

0.540 

1409 

1.527 

0.141 

16 

0.698 

3549 

1.974 

0.  140 

17 

0.816 

4394 

2.308 

0.131 

18 

0.909 

4735 

2.571 

0.123 

19 

0.986 

4862 

2.788 

0.119 

20 

1.052 

4932 

2.976 

0.114 

21 

1.114 

4962 

3.151 

0.  Ill 

22 

1.169 

4983 

3.308 

0.110 

23 

1.225 

4988 

3.464 

0.109 

24 

1.277 

4994 

3.613 

0. 109 

25 

1.332 

4997 

3.766 

0.109 

26 

1.389 

4996 

3.928 

0.111 

27 

1.449 

5000 

4.098 

0.114 

28 

1.517 

4998 

4.291 

0. 120 

29 

1.597 

5000 

4.  517 

0.127 

30 

1.701 

5000 

4.810 

0.146 

31 

1.874 

5000 

5.300 

0.194 
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TABLE  10-  CHANGES  IN  RESULTS  AS  A  FUNCTION  OF  THE 
SIZE  OF  THE  EXPERIMENT 


(a) 


NORMAL  PLOT  DATA 

SITUATION:  n«  32,  k«  31,  r-  8+0-,  d«1.00 


CONTRAST 
-0.686 
-0.525 
0.430 


I  R 
0 


STANDARDIZED 
CONTRAST 
-1.940 
-1.48S 
-1.215 


STD. 

DEVIATION 
0.182 
0.138 
0.121 


4 

-0.359 

1 

-1.014 

0.112 

5 

-0.299 

S 

-0.847 

0.106 

6 

-0.248 

6 

-0.700 

0. 101 

7 

-0.202 

4 

-0.570 

0.098 

8 

-0.158 

9 

-0.446 

0.095 

9 

-0.117 

10 

-0.331 

0.094 

w 

tv34 

10 

-0.078 

14 

-0.220 

0.093 

11 

-0.039 

30 

-0.111 

0.092 

A  . 

12 

-0.001 

28 

-0.004 

0.091 

\\ 

13 

0.036 

40 

0.102 

0.091 

14 

0.074 

62 

0.209 

0.092 

r; 

15 

0.114 

83 

0.323 

0.092 

16 

0.154 

109 

0.436 

0.092 

17 

0.197 

156 

0.558 

0.094 

18 

0.242 

206 

0.684 

0.096 

V 

19 

0.290 

258 

0.820 

0.099 

20 

0.343 

444 

0.971 

0. 102 

r-v 

21 

0.403 

670 

1.139 

0.106 

22 

0.  472 

1050 

1.335 

0.111 

23 

0.555 

1754 

1.568 

0.119 

7*  .  * 

L- 

24 

0.652 

2781 

1.843 

0.131 

r--' 

25 

0.755 

3694 

2.135 

0.138 

> 

26 

0.857 

4288 

2.423 

0. 141 

Si 

27 

0.956 

4618 

2.704 

0.143 

28 

1.0S8 

4810 

2.993 

0.146 

Sa 

29 

1.170 

4918 

3.308 

0.153 

30 

1.303 

4959 

3.684 

0.171 

31 

1.506 

4993 

4.260 

0.218 

(b) 


SITUATION:  n» 


NORMAL  PLOT  DATA 
64,  k-  63,  f  8+0-,  d-1.00 


1 

2 

3 

4 

5 

6 

7 

8 
9 

10 

11 

12 

13 

14 

15 

16 

17 

18 

19 

20 
21 
22 

23 

24 

25 

26 

27 

28 

29 

30 

31 

32 

33 

34 

35 

36 

37 

38 

39 

40 

41 

42 

43 

44 

45 

46 

47 

48 

49 

50 

51 

52 

53 

54 

55 

56 

57 

58 

59 

60 
61 
62 
63 


0.057 
0.069 
0.081 
0.093 
0.105 
0.118 
0.131 
0.143 
0.157 
0.171 
0.186 
0.  201 
0.218 
0.235 
0.253 
0.273 
0.294 
0.317 
0.  344 
0.  376 
0.  415 
0.  466 
0.  539 
0.682 
0.795 
0.885 
0.966 
1 . 040 
1.120 
1.  214 
1.354 


61 


I  R 


STANDARDIZED 

CONTRAST 


-0.570 

0 

-2.281 

-0.473 

0 

-1.893 

-0.419 

0 

-1.675 

-0.378 

0 

-1.512 

-0.346 

0 

-1.383 

-0. 318 

0 

-1.274 

-0.294 

0 

-1.176 

-0.272 

0 

-l. 089 

-0.252 

0 

-1.009 

-0.234 

0 

-0.935 

-0.217 

0 

-0.867 

-0.201 

0 

-0.804 

-0.186 

0 

-0.743 

-0.171 

0 

-0.684 

-0.157 

0 

-0.627 

-0.144 

i 

-to.  574 

-0.130 

0 

-0.522 

-0.117 

0 

-0.470 

-0.105 

0 

-0.421 

-0.093 

0 

-0.371 

-0.081 

0 

-0.324 

-0.069 

0 

-0.278 

-0.058 

1 

-0.230 

-0.046 

0 

-0.184 

-0.034 

0 

-0.137 

-0.023 

0 

-0.090 

-0.011 

0 

-0.044 

0.000 

0 

-0. 001 

8.811 

0 

0.044 

0.  023 

0 

0.090 

0.034 

1 

0.136 

0.045 

0 

0.182 

3 

0 

1 

0 

4 

0 

4 

6 

4 

2 

6 

7 

8 
6 

12 

21 

19 

43 

62 

105 

161 

354 

942 

3732 

4645 

4895 

4967 

4993 

4997 

4998 
5000 


0. 

0.276 
0.322 
0.370 
0.420 
0.471 
0.523 
0.573 
0.627 
0. 685 
0.744 
0.806 
0.873 
0.940 
1.013 
1.091 
1.176 
1.267 
1.376 
1 . 505 
1.659 


1.862 
2.158 
2.729 
3.  1  82 
3.  540 
3.863 
4.160 
4.479 
4.856 
5.415 


STD. 

DEVIATION 
0.114 
0.082 
0.071 
0.064 
0.060 
0.058 
0.055 
0.053 
0.051 
0.050 
0.048 
0.047 
0.046 
0.046 
0.045 
0.045 
0.045 
0.045 
0.044 
0.044 
0.044 
0.043 
0.043 
0.043 
0.042 
0.042 
0.042 
0.042 
0.042 
0.043 
0.043 
0.042 
0.043 
0.043 
0.043 
0.043 
0.043 
0.044 
0.044 
0.045 
0.045 
0.046 
0.047 
0.047 
0.049 
0.050 
0.051 
0.053 
0.055 
0.057 
0.061 
0.064 
0.070 
0.079 
0.093 
0.  11C 
0.115 
0.109 
0.106 
0.107 
0.112 
0.123 
0.151 
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TABLE  10.  (cont’d) 


To  make  the  comparison,  let  us  examine  the  R-spillover  in  the  first 
intended  error  rank  adjacent  to  the  rank  of  the  eight  positive  real  effects. 
The  results  are  shown  in  Table  11. 


TABLE  11.  STANDARDIZED  CONTRAST  DEGRADATION  AND  R-SPILLOVER 
IN  FIRST  ERROR  RANK  AS  A  FUNCTION  OF  n  WHERE  r  =  8  and  d,  =  +1.00O 


k  =  n  -  1 

31 

63 

127 

1st  Error  Rank 

23 

55 

119 

R-Spillover  as  1st  Error  Rank 

Number  of  Intended  Error  Ranks  Into 

34% 

18% 

2% 

Which  Real  Effects  Might  Fall  More 
Than  5%  of  the  Time 

5 

2 

0 

Mean  Standardized  Contrast 

1  .568 

2.158 

2.569 

Expected  Value 

1.929 

2.285 

2.635 

Degradation 

19% 

6% 

2% 

R-Spil lover 

34% 

18% 

2% 

SUMMARY  OF  NORMAL  PLOT  CHARACTERISTICS  AS  A  FUNCTION  OF  N.  K,  R,  R+,  AND  D. 

It  is  apparent  from  the  above  data  that  contrast  degradation  and  R-spillover 
are  functions  of  all  five  parameters,  n,  k,  r,  R+,  and  d.  In  that  regard,  the 
results  of  any  experiment  are  unique. 

This  suggests  that  preparing  a  set  of  generalizable  tables  would  be  a 
horrendous  task  and  probably  far  too  bulky  to  be  used  effectively,  if  at  all. 
The  search  for  the  ideal  solution,  that  is,  to  find  a  simple  mathematical 
equation  which  expresses  these  relationships,  is  beyond  the  scope  of  this 
paper. 

In  Figure  4,  however,  a  two-dimensional  plot  is  given  relating  number  of 
real  effects  and  sizes  of  real  effects  (when  all  are  equal)  to  R-spillover  for 
n  =  32  and  k  =  31.  The  results  are  surprisingly  clean  and  simple,  with  a  few 
exceptions,  possibly  due  to  the  unreliability  of  the  numbers  and/or  an 
inadequate  model  of  the  relationship.  Work  is  ongoing  to  study  this 
relationship  further 


NORMAL  VERSUS  HALF-NORMAL  PLOT  DATA 


In  a  screening  experiment,  the  primary  concern  Is  to  detect  all  of  the 
critical  factors.  Whether  these  factors  have  large  or  small  effects  Is  of 
minor  Importance  In  this  Initial  phase,  and  Interactions  are  only  Important  to 
the  extent  that  some  critical  ones  may  contain  factors  that  do  not  otherwise 
show  up  as  a  critical  main  effect.  That  class  of  Interaction,  an  "intrinsic" 
one.  Is  of  particular  interest.  During  the  screening  phase,  the  Investigator 
will  have  to  weigh  the  cost-effectiveness  of  Including  some  false  positives  In 
order  not  to  miss  some  marginal  real  effects. 

Is  there  a  practical  difference  between  normal  and  half-normal  plots  as  an 
aid  In  the  detection  of  real  effects?  Do  they  differ  in  the  degree  of 
R-splllover  that  occurs  at  ranks  that  fall  at  the  border  of  real  and  error 
contrasts?  There  are  many  who  continue  to  use  the  half-normal  plot  as  a 
detection  device.  Daniel  (1976),  on  the  other  hand,  has  more  recently 
expressed  a  preference  for  the  normal  plot,  although  he  tended  to  use  it  more 
as  a  tool  to  detect  abnormalities  in  his  experimental  data.  This  change 
occurred  because  he  felt  that  the  signed  contrasts  of  the  normal  plot  provided 
more  information  than  the  absolute  contrasts  of  the  half-normal  data. 

In  this  section  we  will  compare  some  advantages  and  disadvantages  of 
normal  and  half-normal  plotting  insofar  as  the  screening  process  is  concerned. 

R-SPILLOVER.  The  data  for  the  following  half-normal  situations,  an  aggregate 
of  5000  runs,  are  shown  In  Table  12. 

n  =  32,  k  =  31,  r  =  8,  d  =  +1.00O;  +1.250;  +1.50;  or  2.000 

8 

n  =  64;  k  =  63;  r  =  8,  d  =  +1.00o,  +1.250,  or  1.50o 

8 

n  =  128,  k  =  127,  r  =  8,  d  =  +1.000 

a 

In  Table  13,  the  R-splllover  patterns  of  these  half-normal  situations  are 
compared  with  those  of  comparable  normal  situations  (data  in  which  the  effects 
were  all  located  at  the  positive  end  of  the  scale). 


6S 


'  J  .  ■••*'/  V 

mJK  mJ* L  * 


TABLE  12.  HALF-NORMAL  PLOT  DATA 
(Situation:  n  =  32,  k  *  31,  r  =  8,  d  =  l.OOo) 


(a) 


<• 

BIAS 

••EFFECTS 

STD  DEV 

STD  BIAS 

1 

0  0187 

30 

0. 01857 

0.  0529 

2 

0. 0375 

37 

0. 02560 

0.  1061 

3 

0  0567 

36 

0. 03084 

0. 1604 

4 

0  0757 

43 

0. 03519 

0. 2140 

5 

0  0951 

38 

0. 03916 

0. 2689 

A 

0.  1149 

45 

0.  04236 

0.  3250 

7 

0  1344 

41 

0. 04487 

0. 3800 

8 

0. 1548 

61 

0. 04770 

0. 4377 

9 

0  1764 

74 

0. 05071 

0. 4989 

10 

0. 1982 

66 

0. 05351 

0.  5605 

1  1 

0.  2203 

91 

0. 05575 

0. 6231 

12 

0.  24  : 

94 

0. 05802 

0. 6880 

13 

0  266 a 

142 

0. 06021 

0.  7546 

14 

0.  2916 

148 

0. 06252 

0. 0247 

15 

0  3175 

170 

0. 06554 

0.  8981 

1  A 

0  3456 

212 

0.  06773 

0. 9776 

17 

0. 3751 

283 

0.  07081 

1. 0610 

ie 

0  4073 

348 

0. 07299 

1.  1519 

19 

0  4425 

504 

0. 07649 

1.  2515 

20 

0.  4815 

609 

0.  08049 

1. 3620 

21 

0  5260 

840 

0.  08611 

1. 4878 

22 

0  5753 

1164 

0.  09094 

1.  6272 

23 

0.  6342 

1610 

0.  09968 

1.  7937 

24 

0  7060 

2443 

0.  10966 

1.  9968 

25 

0. 7875 

3190 

O.  12126 

2. 2274 

26 

0. 8740 

3855 

0.  12926 

2. 4720 

27 

0  9661 

4380 

0.  13858 

2.  7325 

28 

1. 0618 

4669 

0.  14397 

3. 0031 

29 

1.  1717 

4844 

0.  15569 

3.  3142 

30 

1 .  3036 

4947 

0  17253 

3.  6873 

31 

1.  4984 

4986 

0.  20921 

4. 2381 

(b) 


« 

BIAS 

•EFFECTS 

STD  DEV 

STD  BIAS 

l 

0.  0184 

1 

0.  01784 

0.  0522 

2 

0. 0368 

3 

0. 02481 

0. 1042 

3 

0.  0558 

3 

O. 03029 

0. 1579 

4 

0.  0752 

5 

0. 03467 

0. 2128 

5 

0. 0945 

2 

0.  03823 

0. 2674 

6 

0.  1144 

8 

0. 04107 

0. 3234 

7 

0. 1338 

3 

0. 04391 

0. 3784 

8 

0.  1539 

4 

0. 04706 

0.  4354 

9 

0. 1753 

7 

0. 04972 

0.  4958 

10 

0.  1970 

1 1 

0. 05267 

0.  5571 

1 1 

0.  2192 

12 

0. 05551 

0. 6200 

12 

0. 2420 

26 

0. 05756 

0. 6844 

13 

0.  2665 

25 

0.  06010 

0. 7539 

14 

0.  2926 

37 

0. 06221 

0. 8275 

15 

0.  3200 

37 

0. 06530 

0. 9051 

16 

0. 3500 

64 

O. 06788 

0. 9900 

17 

0. 3813 

72 

0. 07128 

1. 0784 

18 

0. 4158 

82 

0. 07502 

1. 1762 

19 

0.  4541 

125 

0. 07988 

1. 2843 

20 

0.  4981 

231 

O. 08555 

1.  4088 

21 

0. 5515 

375 

O. 09194 

1. 5597 

22 

0.  6171 

640 

0. 10072 

1.  7453 

23 

0.  7069 

1251 

O. 11554 

1.  9993 

24 

0. 8390 

3178 

0. 14162 

2.  3731 

25 

0.  9700 

4252 

O. 15021 

2. 7435 

26 

1.  0B65 

4714 

0.  14978 

3. 0732 

27 

I.  1993 

4897 

0. 15087 

3. 3920 

28 

1 .  3060 

4951 

O. 15447 

3. 6940 

29 

1.  4198 

4987 

0. 15757 

4. 0157 

30 

l. 5516 

4997 

0. 17200 

4. 3886 

31 

1. 7483 

5000 

O. 20577 

4.  9450 
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TABLE  12  (cont’d) 

(Situation:  n  =  64,  k  ■  63,  r  M  8,  d  =  l.OOo) 


(c) 


• 

BIAS 

•EFFECTS 

STD  DEV 

STD  BIAS 

1 

O  0189 

0 

0. 01797 

0. 0536 

2 

0. 0377 

0 

0.  02518 

0. 1067 

3 

0.  0566 

0 

0.  03012 

0.  1600 

4 

0  0760 

0 

0.  03445 

0.  2149 

5 

0  0956 

1 

0. 03822 

0. 2703 

6 

0  1155 

0 

0. 04144 

0. 3267 

7 

0. 1358 

0 

0. 04431 

0. 3841 

8 

0. 1563 

1 

0. 04725 

0. 4420 

9 

0  1771 

2 

0. 04955 

0. 5009 

10 

0. 1991 

1 

0. 05185 

0. 5630 

11 

0. 2218 

0 

0. 05458 

0.  6272 

12 

0. 2452 

0 

0.  05747 

0. 6935 

13 

0. 2701 

2 

0. 06044 

0. 7639 

14 

0.  2956 

2 

0. 06334 

0. 8360 

15 

0. 3230 

3 

0. 06618 

0.  9135 

16 

0  3519 

7 

0.  06906 

0.  9954 

17 

0. 3839 

10 

O.  07301 

1.  0859 

18 

0. 4196 

13 

0.  07722 

1. 1868 

19 

0. 4590 

20 

O. 08193 

1 .  2983 

20 

0  5055 

36 

O.  08948 

1.  4297 

21 

0  5637 

88 

0. 09756 

1. 5944 

22 

0. 6395 

190 

0.  11271 

1 .  8088 

23 

0.  7574 

620 

0.  13464 

2. 1421 

24 

l.  0225 

4217 

O.  17780 

2. 8919 

25 

1  1959 

4842 

O.  16831 

3.  3826 

26 

1 .  3322 

4953 

0.  15874 

3. 7682 

27 

1.  4455 

4993 

0.  15461 

4. 0884 

28 

l. 5544 

5000 

0.  15295 

4.  3965 

29 

1 . 6672 

5000 

0.  15986 

4.  7135 

30 

1 .  7989 

4999 

0.  16929 

5.  0880 

31 

1 .  9989 

5000 

0.  20974 

3.  6338 

(d) 


• 

BIAS 

•EFFECTS 

STD  DEV 

STD  BIAS 

1 

0.  0190 

0 

0. 01803 

0.  0338 

2 

0.  0383 

0 

0. 02311 

0. 1083 

3 

0.  0371 

0 

0. 03006 

0. 1616 

4 

0.  0761 

0 

0. 03424 

0. 2152 

5 

0.  0952 

0 

0. 03812 

0. 2694 

6 

0.  1156 

O 

0. 04118 

0. 3270 

7 

0. 1354 

0 

0. 04404 

0.  3831 

8 

0  1562 

0 

0. 04749 

0. 4419 

9 

0.  1770 

0 

0. 04978 

0.  5007 

10 

0. 1990 

0 

0. 05253 

0.  3629 

11 

0.  2220 

0 

0. 05304 

0. 6279 

12 

0. 2455 

0 

0. 03771 

0. 6945 

13 

0. 2698 

0 

0. 06009 

0. 7631 

14 

O. 2958 

0 

0. 06246 

0. 8367 

15 

0.  3240 

0 

0.  06528 

O. 9164 

16 

0  3532 

o 

0.  06859 

0. 9990 

17 

O.  3850 

0 

0.  07233 

1. 0891 

18 

0.  4210 

0 

0.  07738 

1.  1909 

19 

0  4611 

0 

0.  08227 

1. 3041 

20 

0. 3077 

0 

0.  08884 

1.  4359 

21 

0. 3660 

0 

0.  09842 

1.  6009 

22 

0. 6460 

3 

0.  11577 

1.  8272 

23 

0. 7790 

27 

0.  15223 

2.  2034 

24 

l . 3027 

4972 

0.  20994 

4.  2503 

25 

l .  7000 

4999 

0.  17438 

4.  8084 

26 

1 .  8332 

3000 

0.  16083 

3.  1830 

27 

1.  9478 

3000 

0.  13484 

3.  3093 

28 

2. 0376 

3000 

0.  13313 

3.  8198 

29 

2.  1714 

3000 

0.  13927 

6.  1416 

30 

2  3084 

4999 

0.  17177 

6.  3291 

31 

2. 3033 

3000 

0. 20710 

7.  OBIO 
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TABLE  12  (cont’d) 

(Situation:  n  =«  128.  k  =«  127,  r  -  8,  d  =■  1.00O) 


(e) 


« 

CONTRAST 

1 

0.  0036 

2 

0.  0113 

3 

0  0169 

4 

0. 0227 

5 

0. 0284 

6 

0. 0340 

7 

0. 0399 

8 

O.  0457 

9 

0.  0514 

10 

0. 0571 

1 1 

0. 0629 

12 

0  0687 

13 

0. 0747 

14 

0. 0807 

15 

0. 0866 

16 

0.  0927 

17 

O. 0987 

18 

0.  1049 

19 

0.  1111 

20 

0. 1174 

21 

O.  1237 

22 

0. 1301 

23 

0. 1367 

24 

0  1433 

25 

0  1300 

26 

0  1369 

27 

0. 1639 

28 

0.  1710 

29 

0  1782 

30 

0.  1834 

31 

0. 1932 

32 

0. 2009 

33 

0  2087 

34 

0  2168 

33 

0  2230 

36 

0  2336 

37 

0  2423 

38 

0  2313 

39 

0  2610 

40 

0  2709 

41 

0  2813 

42 

0  2919 

43 

0  3031 

44 

0  3150 

#R 

STD.  OEV. 

0 

0.  00533 

0 

0.  00769 

0 

0.  00952 

0 

0.  01084 

0 

0.  01195 

0 

0.  01301 

0 

0.  01397 

0 

0. 01500 

o 

0.  01573 

o 

0.  01652 

0 

0.  01719 

0 

0.  01801 

0 

0.  01860 

0 

0.  01911 

1 

0.  01975 

o 

O.  02029 

1 

0.  02077 

1 

0. 02135 

0 

0. 02196 

0 

O. 02240 

o 

O.  02294 

0 

0. 02338 

0 

0. 02395 

1 

0. 02440 

2 

0. 02491 

0 

0. 02541 

1 

0. 02600 

2 

0. 02625 

2 

0.  02676 

2 

0. 02716 

2 

0. 02766 

1 

0.  02809 

3 

0. 02879 

1 

0. 02940 

3 

0. 02987 

3 

0. 03044 

6 

0. 03100 

8 

0. 03143 

13 

0. 03216 

3 

0  03293 

7 

0  03377 

16 

0  03479 

’  4 

0  03557 

15 

0  03677 

STANDARDIZED 
CONTRAST 
0.  0222 
0. 0432 
0. 0678 
0.  0908 
0.  1136 
0.  1362 
0. 1596 
0. 1826 
0.  2033 
0.  2286 
0.  2514 
0.  2730 
0. 2988 
0. 3228 
0. 3463 
0.  3707 
0. 3948 
0. 4195 
0. 4445 
0. 4697 
0.  4947 
0. 5203 
0. 5467 
0.  5732 
0. 3999 
O. 6275 
0. 6557 
0.  6841 
0.  7129 
0.  7417 
0.  7729 
0.  8035 
0.  8348 
0.  8670 
0.  9000 
0.  9344 
0.  9691 
1.  0039 
1.  0438 
1.  0833 
1  1231 

1  1676 

1  2126 
1  2600 


45 

0. 3276 

46 

0.  3412 

47 

0  3338 

48 

0. 3717 

49 

0.  3892 

30 

0. 4080 

51 

0.  4301 

52 

0.  4554 

53 

0  4880 

34 

0.  3291 

55 

0.  3888 

56 

0.  6971 

57 

0. 7961 

58 

O.  8840 

59 

0.  9632 

60 

1 .  0380 

61 

l. 1194 

62 

1. 2153 

63 

1 . 3350 

'  21 

0.  03803 

1 

33 

O.  03941 

1 

22 

0.  04061 

1 

43 

0.  04243 

1 

36 

0.  04420 

1 

82 

0.  04644 

1 

76 

0.  04917 

1 

161 

0.  03249 

1 

288 

O.  03713 

1 

498 

0.  06453 

2 

1037 

0.  07776 

2 

3405 

0.  10437 

2 

4416 

0.  11008 

3 

4824 

0.  10878 

3 

4949 

0.  10717 

3 

4985 

0.  10890 

4 

4991 

0.  11316 

4 

4999 

0.  12284 

4 

3000 

O.  13014 

3 

3104 

3649 

4231 

4867 

3367 

6320 

7203 

8217 

9322 

1164 

3352 

7886 

1845 

3360 

8526 

1519 

4777 

8613 

4200 
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(Situation:  n 


TABLE  12  (cont’d) 

■  128.  k  -  127,  r  a 


8,  d  =*  l.OOo) 


*  • 


(f) 


v.; 

v. 

k.\ 


k.- 

N 

I*  . 


« 

1 

2 

3 

4 

3 
6 

7 

8 
9 

10 
l  1 
12 

13 

14 

15 

16 

17 

18 

19 

20 
21 
22 

23 

24 
23 
26 

27 

28 

29 

30 

31 

32 

33 

34 
33 

36 

37 

38 

39 

40 

4  l 

42 

43 

44 


CONTRAST 
0  0033 
0  0113 
O.  0169 
0  0223 
0.  0282 
0  0338 
O  0396 
0  0455 
0.  0513 
0  0572 
0629 
0689 
0747 
0808 
0868 
O  0928 
O  0989 
O  1031 
0.  1114 
1178 
1242 
1307 

1373 
1439 
1306 

1374 
1643 
1712 
1782 
1836 
1930 
2006 
2086 
2170 
2252 
2333 
2423 
2315 
2611 
2713 
2814 
2923 
3033 
3154 


#R 


0 

0 

O 

0. 

o 


o 

o 

o 

0 

0. 

o 

0 

0. 

o 

o 

o 

o 

o 

0 

0 

o 

0 

0 

o 

o 

0 

0 

o 

0 

0 


0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

o 

o 

o 

o 

0 

0 

o 

o 

o 

o 

o 

o 

0 

0 

o 

o 

o 

0 

o 

0 

o 

o 

0 

1 

0 

0 

0 

0 

0 

2 

0 


STD.  DEV. 

0.  00349 
0.  00778 
0.  00930 
0.  01090 
0.  01219 
0.  01324 
0.  01422 
O.  01316 
0.  01597 
O.  01677 
0.  01741 
0. 01802 
0. 01870 
O. 01937 
O.  02003 
O. 02043 
0.  02111 
O.  02161 
0. 02212 
O. 02263 
O.  02312 
0.  02352 
0. 02401 
O. 02449 
O.  02301 
0. 02333 
0. 02613 
0. 02661 
0. 02696 
0. 02739 
. 02803 
. 02840 
0. 02889 
02948 
03007 
03079 
. 03106 
03163 
03239 
03317 
03360 
03437 
03344 
03634 


0. 

0. 


0. 

0. 

0 

0. 

0 

0 

0 

0 

0 

0 

0 


STANDARDIZED 
CONTRAST 
0.  0222 
0.  0432 
0.  0677 
0.  0900 
0.  1127 
0.  1353 
0. 1583 
0.  1819 
0.  2032 
0.  2288 
0.  2517 
0. 2754 
0. 2987 
0. 3230 
0.  3472 
O.  3713 
0.  3957 
O.  4206 
0.  4453 
O.  4711 
0.  4969 
O.  5229 
0.  5491 
0.  3736 
0.  6023 
0. 6297 
O.  6370 
0. 6847 
0. 7130 
0.  7424 
0. 7719 
0.  8025 
0. 8343 
0. 8680 
0.  9009 
0. 9340 
0. 9694 
1 .  0060 
1. 0446 
0831 
1236 
1690 
2142 
2618 


45 

0.  3278 

1 

0. 03754 

1. 3111 

V*. 

46 

0.  3414 

1 

0.  03893 

1.  3633 

"  »  " 

47 

0, 3561 

1 

0. 04022 

1.  4242 

y/- 

48 

0. 3718 

1 

0.  04169 

1. 4871 

wliC 

49 

0. 3896 

2 

0.  04404 

1. 5582 

r#r 

SO 

0. 4093 

4 

0.  04654 

1. 6373 

Pvr- 

51 

0.  4322 

4 

0.  04966 

1. 7287 

52 

0.  4595 

10 

0.  03410 

1. 8379 

53 

0.  4929 

18 

0.  06108 

1.  9714 

54 

0.  5399 

31 

0.  07071 

2.  1594 

55 

0.  6198 

236 

0.  09232 

2.  4790 

:-s-: 

56 

0.  9001 

4698 

0.  13805 

3.  6003 

37 

1 .  0358 

4973 

0.  12001 

4.  1431 

BF 

38 

1.  1290 

4997 

0. 11184 

4.  3161 

39 

1. 2107 

4998 

0. 10891 

4.  8429 

60 

1. 2866 

5000 

0. 10949 

3.  1463 

61 

1.  3678 

5000 

0. 11420 

3.  4712 

62 

1.  4632 

3000 

0. 12412 

3.  8528 

63 

1. 6034 

3000 

0. 13289 

6.  4214 

*  .  *  4 

ii 


'm 


\  »>■ 

t— .  ■* 


a 
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TABLE  12  (cont'd) 

(Situation:  n  =  128,  k  =  127,  r  *  8,  d  ■  1.00a) 


STANDARDIZED 

CONTRAST 

#R 

STD.  DEV. 

CONTRAST 

0  0037 

0 

0. 00546 

0. 0226 

45 

0.  3285 

0 

0  03887 

1.  3139 

0.  0113 

'  0 

0. 00764 

0. 0452 

46 

0. 3421 

0 

0.  03991 

1 .  3683 

0  0171 

0 

0. 00952 

0.  0685 

47 

0.  3566 

0 

0.  04138 

1.  4266 

0.  0229 

0 

0.  01084 

0.  0915 

48 

0. 3727 

0 

0.  04321 

1.  4909 

0.  0287 

0 

0. 01205 

0.  1150 

49 

0. 3903 

0 

O.  04508 

1.  5611 

0. 0344 

0 

0.  01311 

0.  1378 

50 

0. 4100 

0 

0.  04699 

1 .  6399 

0. 0401 

o 

0.  01401 

0.  1603 

51 

0. 4322 

0 

0.  05002 

1.  7288 

0.  0460 

0 

0.  01502 

0.  1839 

52 

0. 4394 

O 

0.  05419 

1.  8377 

0.  0518 

0 

0.  01590 

0.  2072 

53 

0. 4938 

O 

0.  06074 

1.  9752 

0.  0576 

0 

0.  01667 

0.  2303 

54 

0.  3408 

3 

0.  07134 

2.  1630 

0.  0634 

0 

0.  01730 

0.  2536 

35 

0. 6233 

21 

O.  09662 

2.  4938 

0.  0692 

0 

0.  01803 

0.  2767 

56 

1.  1429 

4977 

0.  15192 

4.  3717 

0. 0751 

0 

0.  01868 

0.  3003 

57 

1. 2859 

4999 

0.  12378 

3.  1437 

0.  0810 

0 

0. 01931 

0.  3241 

58 

t.  3813 

5000 

O.  11336 

3.  5253 

0. 0870 

0 

0.  01986 

0.  3481 

59 

1. 4619 

3000 

0.  10999 

3.  8477 

0.  0932 

0 

0. 02034 

0.  3727 

60 

1.  3382 

3000 

0.  11083 

6.  1328 

O. 0992 

0 

0. 02080 

0.  3968 

61 

1. 6200 

3000 

0.  11246 

6.  4800 

0.  1054 

0 

0.  02141 

0.  4217 

62 

1. 7159 

5000 

0.  12419 

6.  8635 

O  1116 

0 

0.  02179 

0.  4464 

63 

X. B397 

5000 

0.  15253 

7.  4387 

0.  1180 

o 

O.  02239 

0.  4718 

O. 1244 

0 

0.  02280 

0.  4975 

0  1309 

o 

0.  02325 

0.  5236 

0.  1376 

0 

0.  02395 

0.  3503 

0.  1442 

0 

0.  02438 

0.  5766 

0. 1309 

0 

0.  02315 

0.  6035 

0. 1378 

o 

0.  02563 

0.  6311 

0. 1647 

0 

O.  02618 

0.  6590 

0. 1719 

0 

0.  02668 

0.  6878 

0. 1790 

0 

0.  02710 

0.  7160 

0. 1864 

0 

0.  02765 

0.  7436 

0.  1941 

0 

0.  02834 

0.  7764 

0  2018 

0 

0.  02900 

0.  8073 

0.  2096 

0 

0.  02964 

0. 8385 

0. 2176 

0 

0.  03031 

0. 8704 

0. 2258 

0 

0.  03076 

0.  9031 

0. 2343 

0 

0.  03132 

0. 9373 

0. 2431 

0 

0.  03180 

0. 9723 

0. 2321 

0 

0.  03258 

1.  0086 

0. 2617 

0 

0.  03344 

1. 0466 

0. 2716 

0 

0.  03422 

1. 0863 

0.  2818 

0 

.  0.  03482 

l. 1271 

0  2924 

0 

0.  03538 

1. 1695 

0. 3036 

0 

0.  03636 

1.  2143 

0. 3155 

0 

0.  03774 

1. 2620 

(Situation:  n 


TABLE  12  (cont’d) 

128,  k  =  127,  r  *  8.  d  *  l.OOo) 


(h) 


STANDARDIZED 

64 

0.  1303 

0 

0. 01324 

0.  7373 

CONTRAST 

#R 

STO.  DEV. 

CONTRAST 

65 

0.  1329 

0 

0.  01333 

0.  7515 

1 

0.  0019 

0 

0.  00188 

0.  0107 

66 

0.  1354 

0 

0.  01340 

0. 7657 

C- 

0.  0038 

0 

0. 00258 

0.  0213 

67 

0.  1378 

0 

0. 01359 

0. 7798 

3 

0.  0056 

0 

0. 00317 

0.  0319 

'  68 

0.  1403 

0 

0. 01371 

0. 7939 

4 

0  0075 

0 

O. 00365 

0.  0425 

69 

0.  1429 

0 

0. 01385 

0. 8081 

5 

0  0093 

0 

0.  00402 

0.  0529 

70 

0.  1454 

0 

0. 01402 

0. 8227 

6 

0.  0112 

0 

O. 00440 

0.  0636 

71 

0.  1481 

o 

0.  01414 

0. 8377 

7 

0  0132 

0 

0. 00478 

0.  0745 

72 

0.  1507 

0 

0.  01427 

0.  8526 

8 

0  0151 

o 

0.  00508 

0.  0851 

73 

0.  1534 

0 

0.  01440 

0.  8677 

9 

0.  0170 

o 

0. 00540 

0.  0959 

74 

0. 1561 

0 

O. 01449 

0. 8833 

10 

0.  0188 

o 

0. 00562 

0. 1065 

75 

0  1589 

0 

O.  01460 

0.  8990 

1 1 

O  0207 

0 

0. 00587 

0.  1170 

76 

0  1616 

0 

0.  01468 

0. 9144 

12 

0.  0225 

0 

O.  00612 

0.  1273 

77 

0.  1644 

0 

0.  01483 

0.  9302 

13 

0. 0244 

0 

0.  00632 

0.  1381 

78 

0  1673 

0 

0. 01493 

0.  9463 

14 

0.  0263 

0 

0.  00652 

0.  1487 

79 

0.  1702 

0 

0.  01511 

0.  9629 

15 

0.  0282 

0 

0. 00674 

0.  1594 

80 

0  1732 

0 

O  01523 

0.  9799 

16 

0  0301 

0 

0.  00697 

0.  1703 

81 

0  1762 

o 

C  01539 

O.  9966 

17 

0  0319 

0 

0.  00716 

0.  1807 

82 

0.  1792 

0 

0. 01558 

1.  0140 

18 

0.  0339 

0 

O.  00734 

0.  1916 

83 

0. 1823 

0 

0. 01562 

l.  0313 

19 

0  0358 

0 

0.  00754 

0.  2023 

84 

0.  1855 

0 

0.  01575 

1.  0494 

20 

0  0377 

0 

0.  00770 

0.  2133 

85 

0. 1888 

0 

0.  01599 

1.  0679 

21 

O. 0396 

0 

0.  00789 

0.  2242 

86 

0.  1921 

0 

0.  01615 

1.  0869 

22 

0.  0416 

0 

0.  00805 

0.  2351 

87 

0. 1955 

0 

0.  01629 

1.  1060 

23 

0.  0435 

0 

0.  00819 

0.  2458 

88 

0. 1989 

0 

0.  01650 

1.  1250 

24 

0. 0454 

0 

0.  00838 

0.  2566 

89 

0. 2024 

o 

0.  01673 

1.  1450 

25 

0. 0473 

0 

0.  00854 

0.  2677 

90 

0  2060 

o 

0  01690 

1.  1654 

26 

0.  0492 

0 

0.  00866 

0.  2785 

91 

0  2097 

o 

0.  01709 

1.  1860 

27 

0. 0512 

0 

0. 00882 

0.  2899 

92 

0.  2135 

0 

0.  01732 

1.  2077 

28 

0  0532 

0 

0.  00897 

0.  3009 

93 

0.  2172 

0 

0.  01765 

1.  2288 

29 

0. 0551 

0 

O.  00914 

0.  3118 

94 

0.  2212 

0 

0.  01790 

1.  2512 

30 

0  0571 

0 

0.  00930 

0.  3228 

95 

0.  2253 

0 

0.  01811 

1.  2742 

31 

0  0590 

0 

0.  00943 

O.  3339 

96 

0.  2295 

0 

O.  01832 

1 .  2980 

32 

0.  0610 

0 

0.  00956 

0.  3451 

97 

0.  2338 

0 

0.  01859 

1.  3227 

33 

0. 0629 

0 

0.  00969 

0.  3560 

98 

0. 2382 

o 

0.  01880 

1.  3477 

34 

0  0650 

0 

0.  00981 

0.  3674 

99 

0. 2429 

0 

0.  01926 

1.  3740 

35 

0.  0669 

0 

0.  00993 

0.  3786 

100 

0  2476 

0 

0.  01961 

1.  4005 

36 

0.  0690 

0 

0.  01006 

0.  3900 

101 

0  2524 

0 

0.  02008 

1.  4279 

37 

0.  0710 

o 

0. .01010 

0.  4014 

102 

0.  2575 

0 

0.  02042 

1.  4569 

38 

0  0730 

0 

0.  01023 

0.  4130 

103 

0.  2629 

0 

0.  02079 

1.  4873 

39 

0.  0751 

0 

0.  01032 

0.  4249 

104 

0. 2685 

0 

0.  02118 

1.  5189 

40 

0. 0771 

0 

0.  01040 

0.  4362 

105 

0. 2744 

0 

0.  02159 

1.  5524 

41 

0. 0792 

0 

.  0.  01054 

0.  4477 

106 

0. 2806 

0 

0.  02215 

1.  5875 

42 

0.  0812 

0 

0.  01065 

0.  4594 

107 

0.  2872 

0 

0.  02248 

1 . 6246 

43 

0.  0833 

0 

0.  01073 

0.  4710 

108 

0. 2941 

0 

0.  02323 

1 .  6638 

44 

0. 0854 

0 

0.  01087 

0.  4831 

109 

0.  3015 

0 

0.  02408 

1.  7058 

45 

0.  0875 

0 

0.  01 102 

0.  4950 

110 

0. 3096 

1 

0.  02495 

1.  7514 

46 

0  0896 

0 

0.  01113 

0.  5069 

1  1  1 

0. 3182 

0 

0.  02592 

1 .  8002 

47 

0  0917 

0 

0.  01 129 

0  5189 

112 

0  3277 

0 

0. 02672 

1.  8537 

46 

0  0939 

0 

0  01139 

0  531  1 

1  13 

0. 3382 

1 

0. 02797 

1.  9133 

49 

0  0960 

0 

0.  01 153 

0  5432 

*1  1  4 

0  3504 

0 

0. 02956 

1.  9022 

50 

0  0982 

0 

0.  01 161 

0.  5553 

1  1  5 

0  3643 

2 

0.  03165 

2.  0609 

51 

0  1004 

0 

0  01  173 

0  5680 

1  16 

0. 3808 

2 

0.  03431 

2.  1541 

52 

0  1026 

0 

0  01186 

0  5806 

1  17 

0  4014 

10 

0. 03794 

2.  2707 

53 

0  1049 

0 

0.  01196 

0  5932 

1  18 

0  4313 

16 

0.  04530 

2.  4396 

54 

0  1071 

0 

0  01205 

0  6058 

<  19 

0  4833 

86 

0.  061B4 

2.  7337 

55 

0  1093 

0 

0.  01220 

0  6182 

120 

0  7493 

4889 

0.  10396 

4.  2385 

56 

0  1115 

0 

0.  01228 

0  6308 

121 

0  8501 

4993 

0.  08733 

4.  8090 

5  7 

0  1  138 

0 

0.  01241 

0  6437 

122 

0  9183 

5000 

0.  08054 

5.  1945 

58 

0  1161 

o 

0.  01250 

0  6568 

123 

0  9753 

5000 

0.  07752 

5.  5170 

59 

0  1184 

0 

0.  01260 

0  6699 

124 

1  0297 

5000 

0.  07798 

5  8249 

60 

0  1207 

0 

0.  01272 

0.  6830 

125 

1.  0855 

5000 

0.  08081 

6  1406 

61 

0  1231 

o 

0.  01282 

0  6964 

126 

1  1529 

5000 

0.  08759 

6.  5216 

62 

0  1255 

0 

0.  01297 

0  7097 

>27 

1  2552 

5000 

0. 10822 

7.  1002 

63 

0. 1278 

0 

0.  01308 

0.  7232  '  1 

TABLE  13.  COMPARING  R-SPILLOVER  OF  NORMAL  AND  HALF-NORMAL  PLOTS 
(5000  runs.  Population  sigma  =  1.00) 


N  =  32.  K  =* 

31,  r  *  8, 

d#-  +1.00O 

1st  error  rank  =  23 

+1.250 

+1.50O 

+2.00 o 

Normal 

33%(20)* 

23% (22) 

10%(23) 

0.3% (23) 

Half-normal 

32%(19) 

25%(21) 

12%(22) 

0.5% (23) 

N  =  64,  K  =* 

63,  r  *  8, 

1st  error  rank  =  55 

Normal 

18%(54) 

4% (55) 

0.3%(55) 

Half-normal 

20%(54) 

5%(55) 

0.4%(55) 

N  =  128,  K  =  127,  r  =  8,  1st  error  rank  =  119 

Normal  2%(119) 

Half-normal  2%(119) 

*  The  number  in  parentheses  is  the  first  rank  with  less  than  10%  R-spillover. 

For  all  practical  purposes,  the  degree  of  R-spillover  for  the  aggregate 
normal  and  half-normal  plot  data  are  not  considerably  different.  While  the 
normal  plot  data  might  appear  to  have  a  slight  overall  edge,  having  a  slightly 
smaller  R-spillover  into  the  intended  error  ranks  and  having  fewer  ranks  to 
inspect  for  real  effects  that  have  at  least  a  10%  chance  of  being  there,  these 
differences  are  likely  to  be  obscured  in  a  single  experiment.  For  the 
aggregate  R-spillover  data,  a  difference  of  1%  or  2%  is  probably  within  the 
accuracy  of  the  data. 


With  normal  plot  data,  we  have  seen  how  the  R-spillover  for  the  same 
number  of  real  effects  is  less  at  the  borders  between  intended  real  and  error 
ranks  when  there  are  both  positive  and  negative  real  effects,  than  is  at  the 
border  when  all  are  positive.  Therefore,  for  the  same  number  of  real  effects, 
the  normal  situation  would  be  expected  to  result  in  a  smaller  R-spillover  than 
the  half-normal  when  positive  and  negative  effects  are  present.  This  means 
that  the  chance  of  selecting  error  contrasts  decreases  with  the  normal  plots. 


ESTIMATING  SIGMA.  To  decide  whether  an  effect  is  real  or  not  with  a  certain  * 
error  rate,  the  population  sigma  must  be  known.  The  population  sigma  can  be 
estimated  by  calculating  the  slope  of  the  standardized  contrasts.  These 
requirements  lead  to  two  circular  situations,  each  a  classic  "Catch  22." 

1.  The  exact  number  of  the  errors  contrast  can  be  known  by  identifying 
which  contrasts  represent  real  effects;  the  remaining  ones,  of  course,  are  the 
error  contrasts.  But  to  determine  how  many  contrasts  are  real,  one  must  have 
an  estimate  of  the  standard  error  of  the  contrasts,  which  is  derived  from 
knowledge  of  the  slope  of  the  error  contrasts.  The  accuracy  of  this  estimate 
decreases  markedly  when  the  exact  number  of  error  contrasts  are  not  known. 

2.  Standardized  values  must  be  used  when  the  slope  of  the  error 
contrasts  used  to  estimate  the  population  sigma.  To  standardize  the  error 
contrasts,  the  correct  estimate  of  the  population  sigma  is  required.  While 
other  ways  have  been  suggested  for  estimating  sigma,  the  slope  of  the 
contrasts  is  still  the  most  accurate. 

As  the  size  of  the  real  effects  diminish  or  the  number  of  real  effects 
increase,  the  R-spillover  also  Increases  and  the  chance  of  determining  the 
exact  number  of  real  effects  and  error  contrasts  also  decreases. 

With  normal  plots,  there  is  an  additional  problem.  We  must  not  only 
decide  how  many  error  contrasts  there  are  but  also  at  what  ranks  they  are 
located.  Failure  in  either  case  means  that  we  are  plotting  the  error 
contrasts  against  the  incorrect  estimated  values  of  order  statistics.  This 
result  is  an  incorrect  slope. 

OVER-  AND  UNDERESTIMATING  THE  NUMBER  OF  ERROR  CONTRASTS.  Let  US  determine 
what  happens  to  our  estimate  of  the  population  sigma  if  we  over-  or 
underestimate  the  number  of  error  contrasts  in  normal  and  half-normal  plot 
data.  Ttie  following  situation  will  be  usel: 

n  12,  k  =31,  r  =  8 ,  d  -  *1.00  [e  ---  k  -  r  =  23) 


TABLE  14.  COMPARING  THE  ROBUSTNESS  OF  NORMAL  AND  HALF-NORMAL  PLOTS 
ON  POPULATION  SIGMA  ESTIMATIONS  WHEN  THE  NUMBER  OF  ERROR 
CONTRASTS  ARE  OVER-  AND  UNDERESTIMATED 


Number  of  Contrasts 
Used  (0.7e) 


Population  Estimates 

Mean  Slope  (Normal)  Mean  Slope  (Half-normal) 


18 

out 

of 

26 

(More) 

0.95s 

[Aver. 

1.05s] 

1.15s 

18 

out 

of 

25 

■  uSm 

17 

out 

of 

24 

16 

out 

of 

23 

(TRUE) 

1.00s 

[Aver. 

1.00s] 

1.00s 

16 

out 

of 

22 

0.93s 

15 

out 

of 

21 

0.88s 

14 

out 

of 

20 

(Less) 

1.14s 

[Aver. 

.99s] 

0.84s 

Note  the  Inverse  relationships  that  exist  between  the  normal  and  the 
half-normal  plot  estimates  when  too  many  or  too  few  error  contrasts  are  used 
to  estimate  the  sigma.  For  the  normal  plot  data,  underestimating  the  number 
of  error  contrasts  lead  to  an  overestimation  of  sigma,  while  with  the 
half-normal  plot  data,  It  leads  to  an  underestimation.  The  data  in  Table  14 
suggests  that  estimates  of  the  population  sigma  from  normal  plot  data  Is 
somewhat  more  robust  than  from  half-normal  plot  data,  at  least  when  the  number 
of  error  contrasts  are  overestimated. 

Since  the  data  can  be  plotted  either  way  once  it  has  been  obtained,  the 
reciprocal  results  in  Table  14  suggest  that  the  best  estimate  of  sigma  can  be 
obtained,  whether  the  number  of  error  contrasts  are  over-  or  underestimated, 
by  averaging  the  estimate  from  the  normal  order  data  with  that  from  the 
half-normal  data.  In  the  above  table,  when  this  was  done  with  the  correct 
answer  being  1.00,  we  obtained  an  average  value  of  1.05  when  18  out  of  26 
error  contrasts  were  used  to  calculate  the  slope  and  0.99  when  14  out  of  20 


WHAT  PROPORTION  OF  THE  ERROR  CONTRASTS  SHOULD  BE  USED?  In  Version  5,  Zahn 
threw  away  the  larger  0.30  of  the  error  contrasts  before  calculating  the  slope 
of  the  standardized  contrasts.  By  noting  how  R-spillover  relates  to  the 
degradation  in  the  error  contrasts  from  the  expected  values,  it  is  apparent 
that  this  rule,  empirically  determined,  is  a  consequence  of  the  R-spillover 
that  occurred.  More  R-spillover  would  be  expected  to  occur  in  data  from  a  n  * 
16  experiment  than  might  be  expected  with  larger  experiments.  In  the 
half-normal  plot,  with  four  real  effects  out  of  15  contrasts,  Zahn’s  typical 
situation,  the  three  largest  contrasts  —  0.30  of  the  11  error  contrasts  — 
would  not  be  included  in  the  calculation  of  his  slope.  Those  contrasts  fall 
at  the  ranks  where  the  greatest  amount  of  spillover  would  ordinarily  occur  for 
moderate-sized  effects.  However,  applying  the  same  proportion  to  a  larger 
design,  e.g.,  n  =  32,  k  =  31,  with  four  real  effects,  the  eight  largest  error 
contrasts  would  not  be  used  out  of  27.  In  view  of  the  data  shown  in  Table  12, 
that  could  be  overly  conservative. 

SINGLE  RUN  DATA 

Up  to  this  point,  we  have  been  examining  aggregate  data,  the  mean  of  5000 
runs.  This  tends  to  provide  a  cleaner  picture  than  one  would  expect  to  find 
in  a  real-world,  single-run  experiment.  In  the  remainder  of  this  section,  we 
will  look  at  some  results  from  individual  runs,  results  taken  at  random  from 
the  5000  runs  of  the  earlier  analyses. 

DETECTION.  In  Figures  5  through  8,  each  normal  plot  represents  a  single  run 
that  was  purposefully  selected  —  to  illustrate  a  particular  point  —  from  a 
set  of  50  which  the  computer  had  randoraally  selected  from  the  complete  set  of 
5000  runs.  The  situations  from  which  these  plots  were  selected  are: 


Figure 

5: 

n  =  32, 

k  =  31, 

r 

=  8+ ,  d  =  l.OOo 

Figure 

6: 

n  =  32, 

k  =  31, 

r 

=  8+.  d  =  1.25o 

Figure 

7: 

n  =  32, 

k  =  31, 

r 

=  5+ , 3- ,  d  =  +1.25o 

Figure 

8: 

n  ---  32, 

k  =  31, 

r 

=  8+,  d  =  +1.67 a 

In  both  Figures  5-a  and  5-b,  the  effects  in  the  first  seven  ranks  appear 
sufficiently  off  the  error  line  to  be  considered  real.  However,  this  is 
incorrect.  For  this  set  of  data,  the  actual  locations  of  the  real  effects  had 
been  tracked.  In  fact,  in  Figure  5-a,  the  plot  seventh  from  the  largest  was 
actually  an  error  contrast  and  the  plots  in  the  eighth  and  seventeenth  ranks 
were  the  remaining  two  real  effects  (R-spillover) .  In  Pigure  5-b,  however, 
there  were  no  Inversions  of  real  and  error  contrasts;  the  contrast  in  the 
eighth  rank  is  actually  real,  although  this  would  not  have  been  detected  by 
observing  these  plots.  In  Figure  5-c,  there  appear  to  be  nine  real  effetts, 

although  in  fact  there  are  only  eight;  the  contrast  in  the  sixth  rank  is 

actually  an  error  contrast  (E-spillover).  In  Figure  5-d,  it  is  almost 
impossible  to  determine  how  many  real  effects  might  be  present.  Actually,  the 
eighth  rank  is  an  error  contrast  and  the  ninth  rank  is  the  last  real  effect. 

In  Figure  6,  with  an  effect  of  more  moderate  size,  d  =  1.25o,  we  see  a 
plot  (Figure  6-a)  in  which  all  eight  might  be  detected,  although  the 
transition  from  real  to  error  is  not  sharp  (which  could  conceivably  be  the 
result  of  an  inversion  of  real  and  error  effects).  In  Figure  6-b,  only  six  of 

the  eight  real  effects  are  distinguishable.  In  Figure  6-c.  a  ninth  additional 

real  positive  effect  apparently  stands  out  and  there  might  be  one  real 
negative  effect.  Since  the  tracking  capability  had  not  been  written  into  the 
simulation  program  when  these  plots  were  generated,  we  have  no  idea  to  what 
extent  these  apparent  interpretations  are  correct. 

In  Figure  7,  we  also  have  eight  real  effects,  d  *  1.25,  but  five  of  them 
are  positive  and  three  negative.  How  does  this  show  up  on  the  plots?  With 
normal  plots  and  with  the  e.v.n.o.s.  on  the  ordinate,  the  positive  real 
effects  should  fall  off  below  the  line  and  the  negative  real  effects  should 
fall  off  above  the  line.  Falling  on  the  opposite  sides  indicates  that  the 
data  may  be  truncated.  From  the  plots  in  this  figure,  it  can  be  seen  that 
sometimes  they’re  all  apparently  clearly  visible  (Figure  7-a),  and  sometimes 
none  of  them  are  (Figure  7-b).  Much  of  the  time,  detectability  falls  in 
between,  for  example  In  Figure  7-c,  with  the  five  positive  ones  being 
questionable,  and  the  three  negatives  being  more  clearly  visible. 
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Figure  5-b.  Sample  Individual  Normal  Plot  for  Situation: 
n=32,  k=31,  r=8+,  d  =  l.OOd 
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Figure  5-c.  Sample  Individual  Normal  Plot  tor  Situation 
n=32,  k=31,  r=8+,  d  ;  l.OOo 
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Figure  6-a.  Sample  Individual  Normal  Plot  for  Situation: 
n=32,  k=31,  r=8+,  d  =  1.250 
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Finally,  in  Figure  8,  we  can  see  how  relatively  large  real  effects 
influence  Individual  plots.  This  still  does  not  guarantee  that  there  will  not 
be  some  plots  that  are  so  ambiguous  that  not  one  of  the  eight  effects  stands 
out  clearly  (Figure  8-a).  On  the  other  hand,  they  may  stand  out  quite  clearly 
(Figure  8-b),  or  they  may  vaguely  stand  out,  but  with  a  slow  transition  rather 
than  a  distinct  break  between  real  and  error  contrasts  (Figure  8-c).  This 
last  example  makes  one  suspect  there  will  be  some  inversions  between  real  and 
error  contrasts. 

ESTIMATING  SIGMA.  In  the  aggregate  data,  estimates  of  the  population  sigma 
from  slope  calculations  frequently  produced  satisfactory  results.  But  those 
were  based  on  the  means  of  5000  runs.  How  good  might  these  estimates  be  for  a 
single  run? 

This  situation  was  investigated: 

n  »  32,  k  *>  31,  r  =  8,  r*  -  8,  d  »  1.25o. 

With  eight  real  effects,  there  are  23  error  contrasts.  For  this  analysis,  all 
error  contrasts  were  at  the  minus  end  of  the  continuum  of  a  normal  plot.  A 
slope  was  calculated  for  the  15  smallest  standardized  error  contrasts  out  of 
the  23  against  their  corresponding  ranks  of  the  e.v.n.o.s.  for  k  *  23.  These 
15  approximate  the  0.70  part  of  the  error  contrasts  that  Zahn  proposed  should 
be  used.  The  standardized  error  contrasts  were  used  for  the  calculation  since 
this  enabled  the  slope  to  be  the  appropriate  estimate  of  the  population  sigma, 
in  this  case  equal  to  1.00. 

In  Figure  9,  a  histogram  is  shown  for  50  slopes  (each  from  an  individual 
run),  selected  at  random  from  5000  runs  on  the  above  situation.  The  mean 
slope  of  all  50  slopes  Is  1.002,  which  Is  equivalent  to  the  true  population 
sigma  of  1.00.  The  standard  deviation  of  these  50  slopes  Is  0.,'05.  That 
means  that  if  the  distribution  Is  essentially  normal,  t>H\  of  t  he  ’.lopes  |  .[  i 
single  run  would  vary  between  O.MO  to  1  .  ,’n  .  in  the  <1 1st  t  Unit  1<>ii  sh.<wn  In 
Figure  11,  the  values  art*  actually  O./M  and  1  .  / 1  t  rspet  t  1  ve  I  y ,  and  t  tie  full 
range  of  Individual  sigmas  for  the  same  situation  r  angrd  t  mm  ().*>.’  to  1.4V 
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This  wide  distribution  Is  not  untypical  of  the  other  distributions  that  were 
calculated  but  not  shown  here.  What  is  important  to  note  is  that  in  a  single 
experiment,  the  estimated  population  sigma  may  be  off  more  than  20\  over  32\ 
of  the  time  strictly  by  chance. 

The  analysis  of  the  above  data,  as  is  the  case  with  so  much  of  the  data  In 
this  report,  took  advantage  of  the  fact  that  we  knew  what  the  correct  sigma 
was  and  how  many  real  and  error  effects  there  really  were.  We  estimated  the 
standardized  contrasts  using  the  correct  value  for  the  square  root  of  four 
over  n,  or  in  this  case,  0.3536.  But,  in  the  real  world,  we  don't  have  this 
perfect  knowledge.  Just  what  effect  does  being  off  on  the  initial  estimate  of 
sigma,  used  to  standardize  our  contrasts,  really  make  in  the  final  estimate  of 
the  population  sigma? 

In  Table  15,  the  mean  slope  (or  final  estimate  of  the  population  sigma) 
tor  5000  runs  is  shown  along  with  its  standard  deviation  when  the  estimated 
sigma  used  to  calculate  the  standardized  values  is  over  and  underestimated. 

The  situation  use!  was: 

n  --  32,  k  =  31,  r  "  9,  d  ’  +1.25  (see  Table  5  t  1 


TABl.K  15.  EPKfcUT  if-  ERRORS  IN  THE  INITIAL  ESTIMATE  OK 
SIGMA  ON  THE  KINAl.  ESTIMATE  OK  SIGMA  (50  tuns) 
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Overestimates  of  the  initial  sigma  of  the  normal  plots  led  to  an 
underestimation  of  the  population  sigma,  on  average,  and  underestimation  in 
the  initial  stage,  to  an  overestimation  of  the  population  sigma.  This 
phenomenon  is  strictly  an  arithmetic  consequence  of  the  fact  that  to  convert 
the  raw  contrast  to  the  standardized  values  used  to  estimate  the  slope,  we 
divided  by  the  initial  estimate  of  sigma  multiplied  by  the  square  root  of  four 
over  n  (see  Equation  6).  This  can  be  illustrated  in  the  following  manner, 
holding  the  square  root  of  four  over  n  constant: 


Initial  Slqma 

Raw  Contrasts 

Standardized  Contrasts 

Slope 

1  (True) 

2,4,6,8,10 

2,4,6,8,10 

2 

2  (Over) 

2,4,6,8,10 

1,2, 3. 4, 5 

1 

0.5  ( Under ) 

2,4,6,8,10 

4,8,12,16,20 

4 

Although  the  standard  deviation  of  the  underestimated  final  sigma  is  smaller 
than  the  overestimation,  the  absolute  differences  from  the  true  sigma  are 
greater . 
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In  Table  16.  the  critical  values  (or  guardrails)  for  a  =  0.05,  0.10, 

C .  20 .  and  0.40  are  given  for  half-normal  plot  data  w4ien  k  *  31  and  63.  These 
n 'ur.be rs  would  be  used  to  test  whether  the  largest  contrast  in  any  set  of  k's, 
is  larger  than  the  critical  value  for  the  given  error  rate,  a.  If  so,  it 
would  be  judged  real. 

If  this  information  is  to  be  applied,  the  investigator  would  first  plot 
his  data  on  a  half  normal  gild  and  remove  all  effects  that  obviously  fall  off 
the  line  formed  by  the  estimated  values  of  order  statistics  for  k.  Sensitive 
to  the  R  spillover  that  is  likely  to  occur,  he  may  examine  one  or  two  ranks 
below  that  point  to  detect  logical  candidates  for  real  effects.  For  example, 
ir  kit*-  :  A  arid  AH  •-•*■:*•  obviously  large  and  Effect  B  was  at  a  borderline 
rink,  it  w-  .11 1-1  tentatively  considered  real.  Once  the  tentative  and  obvious 
real  effe-  ts  have  been  removed,  the  remaining  k  contrasts  will  be  tested  one 
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TABLE  16. 


GUARDRAILS  FOR  HALF-NORMAL  PLOTS  WHERE 
k  a  31  and  63 


k 

.  03 

Va 1 u.« 

.  10 

of  alpha 
.  20 

.  40 

31 

3.  326 
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2.  759 

2.  408 

30 
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29 
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28 
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27 
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3.  490 
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3.  473 
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38 

3.  439 
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at  a  time  against  the  appropriate  values  in  Table  16.  For  example,  if  six 
contrasts  have  already  been  removed  from  an  original  63,  then  we  would  compare 
the  next  contrast  with  the  critical  values  at  rank  57  in  Table  16.  If  it 
exceeds  the  appropriate  value,  then  the  next  lower  contrast  is  examined 
against  the  critical  values  at  rank  56,  and  so  forth. 

The  critical  values  were  obtained  using  a  Monte  Carlo  approach  employing 
10,000  runs.  The  raw  contrasts  are  standardized  using  an  estimate  of  o, 
based  on  the  slope  of  the  smallest  0.85  of  the  contrasts  which  are  tentatively 
assumed  to  be  error  contrasts.  The  number  of  contrasts  from  which  the 
smallest  0.85  are  taken  is  reduced  by  one  each  time  the  critical  values  at  a 
new  rank  are  calculated.  This  accounts  for  the  fluctuation  in  values  seen  in 
the  tables  as  k  decreases. 

At  ?ach  rank,  the  critical  values  were  calculated  for  the  case  where  k  is 
the  highest  rank  for  the  k  contrasts.  Thus,  each  time,  the  test  is  made  on 
only  the  largest  one. 
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The  guardrails  are  not  provided  for  all  ranks,  only  enough  to  test  down  to 
rank  44  when  starting  with  k  *  63,  and  down  to  rank  20  when  starting  with  k  = 
31.  This  allows  us  to  test  for  20  real  effects  when  initially  there  are  63 
contrasts  and  12  real  effects  when  initially  there  are  31  contrasts.  If  more 
critical  values  are  needed,  they  may  be  generated  using  the  computer  program 
for  guardrails  developed  by  Dr.  David  G.  Weinman  given  in  Appendix  G.  That 
program  can  be  modified  to  generate  guardrails  for  normal  as  well  as 
half-normal  plots. 
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It  should  not  be  overlooked  that  the  use  of  guardrails  at  given  alpha 
values  is  essentially  a  test  to  avoid  Type  1  errors  and  the  alpha  indicates 
the  probability  that  a  nonreal  effect  will  be  included.  But  for  screening 
purposes,  it  is  the  Type  II  error  we  wish  to  avoid.  Therefore,  in  using  these 
guardrails,  one  should  probably  use  the  larger  alpha  values,  closer  to  the  PER 
for  the  number  of  contrasts  being  examined;  this  increases  the  chances  that  a 
real  effect  will  not  be  overlooked  while  allowing  more  null  ones  in.  At  the 
same  time,  the  effective  investigator  will  also  inspect  his  data  and 
rationally  evaluate  the  effects  tentatively  considered  to  be  real.  This  helps 
him  determine  which  conditions  will  be  included  in  the  next  block  of  the 

sequential  data  collection. 
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SECTION  VII 

RELEVANT  PAPERS 

A  literature  search  discovered  more  than  100  reports  in  which  Daniel's 
(1959)  and  Zahn's  (1975a,  b)  work  on  half-normal  plots  were  referenced 
(Appendix  A).  Many  of  these  reports  only  describe  how  the  half-normal  plot 
has  been  put  to  use  to  interpret  particular  experimental  data,  generally 
without  the  use  of  sophisticated  criteria  to  establish  significance.  A  few, 
mostly  from  the  statistical  journals,  examined  the  operating  characteristics 
of  these  plots  or  supplied  alternative  approaches  to  some  of  the  problems  of 
interest  in  this  report.  While  not  Intended  to  be  a  complete  selection,  18 
from  that  group  were  chosen  for  review  on  the  basis  of  the  following  criteria: 

1.  It  is  relevant  and  appears  to  have  some  potential  for  improving  the 
methodology  of  holistic  human  performance  experiments. 
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2.  It  is  not  so  overwhelmingly  mathematical  that  its  practical 
methodology  is  doubtful. 

3.  It  illustrates  alternative  ways  to  use  the  normal  or  half-normal  plot. 

4.  It  proposes  modified  techniques  in  lieu  of  the  normal  or  half-normal 
plot. 

5.  It  provides  illuminating  nonmatheraatical  discussions  related  to 
problems  faced  in  this  report. 

6.  It  provides  a  list  of  references,  some  of  which  might  prove  fruitful 
to  review  further. 
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A  short  description  of  the  selected  references  are  reported  below.  It  was 
beyond  the  scope  of  this  project  to  study  these  in  detail  to  determine  if  and 
how  these  techniques  might  be  incorporated,  along  with  the  half-normal  plot, 
into  the  methodology  for  a  holistic  approach  to  human  performance  research. 
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IDENTIFYING  SIGNIFICANT  EFFECTS  IN  UNREPLICATED  EXPERIMENTS 

i 

Box,  G.  B.  P.,  and  Meyer,  R.  D.  An  analysis  for  unrepllcated  fractional 

factorials.  Technometrics.  1986,  28,  11-18. 

In  the  screening  stage  of  industrial  experimentation  it  Is  frequently  true 
that  the  "Pareto  Principle*  applies.  That  principle  states  that,  a  large 
proportion  of  process  variation  is  associated  with  a  small  proportion  of  the 
process  variables,  in  such  circumstances  of  "factor  scarcity,"  unreplicated 
fractional  designs  and  other  orthogonal  arrays  have  frequently  been  effective 
when  used  as  a  screen  for  isolating  preponderant  factors.  A  useful  graphical 
analysis  due  to  Daniel  (1959)  employs  normal  rather  than  half-normal  plotting 
for  that  purpose.  A  more  formal  analysis  is  presented  in  this  paper,  which 
may  be  used  to  supplement  such  plots  and  hence  to  tacilitate  the  use  of  those 
unreplicated  experimental  arrangements. 

The  technique  relies  on  reasonably  accurate  estimates  of  two  parameters; 
the  probability  of  an  active  effect,  a,  and  the  inflation  factor  of  the 
standard  deviation  produced  by  the  active  effect,  x.  The  authors  rely  on 
some  existing  literature  (mainly  from  the  chemical  industry)  to  obtain  an 
approximate  estimate  of  what  these  values  might  be.  Employing  a  Bayesian 
approach,  they  compute  the  posterior  probability  that  an  effect  is  real.  They 
feel  that  by  utilizing  modern  numerical  computing  methods,  the  posterior 
probability  calculations  they  propose  can  rapidly  be  made  to  provide  visual 
displays  of  probability  plots  of  the  type  they  employed.  From  their  analysis, 
they  can  determine  whether  reasonably  reliable  conclusions  regarding  whether 
effects  are  real  are  possible  from  the  existing  data  or  whether  further 
experimentation  is  needed.  They  claim  that  the  "conclusions  drawn  from 
[their]  analysis  are  usually  insensitive  to  moderate  changes  in  a  and  x, 
and  [they]  believe  that  little  would  be  gained  by  attempting  to  be  more 
precise"  (p.  13).  The  extent  to  which  this  technique  as  an  adjunct  to 
Daniel's  plotting  procedure  is  applicable  and  practical  in  human  performance 
research  should  be  investigated  further. 


Box,  0.  B.  P.,  and  Meyer,  R.  D.  Dispersion  effects  from  the  fractional  , 

designs.  Technometrics.  1986,  28,  19-27. 

The  authors  show  how  it  is  sometimes  possible  to  use  unreplicated 
fractional  designs  to  identify  factors  that  cause  variability  in  performance 
as  well  as  changes  in  average  performance. 

Holms,  A.  G.,  and  Berrettoni,  J.  N.  Chain-pooling  ANOVA  for  two-level 

factorial  replication-free  experiments.  Technometrics.  1969,  11,  725-746. 

While  Daniel  concluded  that  the  half-normal  plot  would  be  effective  when 
only  a  few  effects  were  significant,  these  investigations  try  to  develop 
decision  procedures  to  be  used  when  more  effects  are  significant  and  only  a 
very  small  number  of  effects  can  be  used  to  estimate  the  error  variance.  By 
testing  the  2^  -  r  -  1  mean  squares  in  order  of  increasing  magnitude,  they 
use  their  procedure  to  try  and  estimate  the  number  of  null  effects,  which  are 
then  pooled  into  the  denominator  of  the  test  statistic.  The  authors  admit  the 
technique  is  "good,  even  though  it  will  not  be  shown  analytically  to  be 
best."  Appropriate  risk  functions  are  defined  and  several  modifications  of 
the  suggested  procedure  are  evaluated  by  Monte  Carlo  methods  in  terms  of  the 
risk  functions. 


OTHER  APPLICATIONS  FOR  PLOTS 

Barnett,  V.  The  study  of  outliers:  Purpose  and  model.  Journal  of  the  Royal 
Statistical  Society,  Series  C  (Applied  Statistics),  1978,  27,  242-250. 

The  article  discusses  and  illustrates  the  value  of  categorizing  the 
different  reasons  outliers  occur,  the  ways  of  handling  outliers,  and  the 
models  for  outlier-generation.  The  question  is  raised  as  to  whether  outliers 
should  be  removed  "as  alien  contaminants"  or  ignored  until  there  is  overt 
practical  evidence  that  they  are  unrepresentative. 


Hills,  M.  On  looking  at  large  correlation  matrices.  Biometrika.  1969,  56. 
249-253. 

Two  graphical  techniques  are  applied  to  a  correlation  matrix.  The  method 
of  half-normal  plotting  is  used  to  determine  which  coefficients  are 
numerically  too  large  to  have  come  from  zero  population  values.  A  z-transform 
is  used  before  the  absolute  values  of  the  correlated  data  are  ordered  and 
plotted.  A  visual  clustering  method  is  used  also  to  select  clusters  of 
variable  which  have  high  positive  correlations  with  each  other. 

Prew,  R.  D.,  church,  B.  M.  et  al.  Some  factors  limiting  the  growth  and  yield 
of  winter  wheat  and  their  variation  in  two  seasons.  Journal  of  Agricul¬ 
tural  Science.  Cambridge.  1965,  104,  135-162. 

The  authors  use  the  half-normal  plot  to  examine  the  significance  of  56 
three-factor  interactions,  removed  from  the  rest  of  the  experimental  data  (p. 
158-159).  The  square  roots  of  the  individual  interaction  mean  squares  were 
plotted. 

Schweder,  T.,  and  Spjotvoll,  E.  Plots  of  P-values  to  evaluate  many  tests 
simultaneously.  Biometrika.  1982,  69,  493-502. 

By  applying  a  normal-scores  transform  to  both  axes,  a  P- value  is  converted 
to  a  normal  plot.  The  inverse  transformation  may  also  be  carried  out.  The 
properties  of  the  P-value  plot  are  studied  in  some  detail  for  such  problems 
as:  comparing  all  pairs  of  means  in  a  one-way  layout;  testing  all  correlation 
coefficients  in  a  large  correlation  matrix;  and  evaluating  all  2X2  subtables 
in  a  contingency  table.  Using  Hills's  (see  above)  data  for  comparison,  they 
tended  to  prefer  the  P-value  plot  over  the  half-normal  plot  (p.  497-498)  for 
three  reasons:  (1)  The  uniform  probability  transform  (the  P-value)  is  widely 
used  when  testing  hypotheses;  (2)  It  is  slightly  easier  to  evaluate  the 
variance  of  a  P-value  than  of  a  half-normal  plot;  (3)  It  is  somewhat  easier  to 
fit  a  line  to  the  P-value  plot  (because  the  tails  of  both  axes  are  too 
stretched  out  in  the  half-normal  plot). 


Snee,  R.  D.  Graphical  analysis  of  process  variation  studies.  Journal  of 

Quality  Technology.  1983,  15,  76-88. 

Graphical  techniques  for  analyzing  the  results  of  nested  studies  are 
presented  and  Illustrated  with  examples.  In  addition  to  his  "standard 
deviation  control  chart  analysis,"  Snee-  also  briefly  covers  the  use  of  gamma 
probability  plotting  and  data  transformations  (p.  85-88)  to  evaluate  the 
homogeneity  of  a  group  of  variances,  that  Is,  to  determine  whether  they  are 
all  estimates  of  a  single  population  variance.  If  the  variances  are 
homogeneous,  then  they  will  follow  ax*  distribution  which  Is  a  special 
case  of  the  gamma  family  of  distributions.  The  gamma  plot  operates  like  the 
probability  plot.  The  half-normal  distribution  is  a  special  case  of  the  gamma 
distribution  proposed  by  Vilk,  Gnanadesikan,  and  Huyett  (1962). 

VARIATIONS  ON  THE  NORMAL  PLOT,  GRAPHIC  AND  OTHERWISE 


Andrews,  D.  P. ,  and  Tukey,  J.  W.  Teletypewriter  plots  for  data  analysis  can  be 
fast:  6-line  plots,  including  probability  plots.  Journal  of  the  Royal 
Statistical  Society,  Series  C  (Applied  Statistics),  1973,  22,  192. 

Plots  are  generated  which  may  be  produced  very  quickly  on  a  teletypewriter 
or  similar  remote  terminal.  The  methods  are  useful  for  all  displays  which  can 
be  regarded  as  some  form  of  the  inspection  of  residuals.  Versions  for  use  In 
probability  plotting,  as  an  example  not  always  thought  of  as  an  examination  of 
residuals,  are  also  given.  One  technique  (mentioned  in  Daniel,  1959,  p.  317) 

Is  described  which  avoids  the  need  to  fit  a  straight  line:  plot  the  log  of  the 
observed  effect  on  an  axis  at  135°  to  the  axis  on  which  the  log  average 
quantiles  are  plotted  (p.  199-200). 
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Box,  G.  B.  P.,  and  Tiao,  G.  C.  A  Bayesian  approach  to  some  outlier  problems. 

Biomet rlka.  1968,  55,  119-129. 

The  contents  of  this  paper  may  never  be  applicable  to  human  performance 
research  because  the  authors  assume  a  precision  and  set  of  a  priori 
assumptions  that  are  not  likely  to  be  achieved.  The  paper  is  cited  here, 
however,  because:  (1)  A  nonquant itative  Bayesian-like  approach  underlies  the 
holistic  approach  to  human  performance  research;  (2)  This  paper  is  a 
precursor  to  the  Box  and  Meyer  paper  cited  earlier;  (3)  Anything  Box  writes 
should  be  understood,  even  if  it  isn't  immediately  applicable. 


Margolin,  B.  H.  Systematic  methods  for  analyzing  2  3  factorial  experi¬ 

ments  with  applications.  Technometrics.  1967,  9,  245-259. 

Two  systematic  procedures  to  facilitate  the  analysis  of  a  complete  2n 
m 

3  factorial  experiment  are  discussed.  The  methods  are  applicable  when  all 
quantitative  three-level  factors  are  equally  spaced  and  when  the  contrasts 
involving  qualitative  three-level  factors  appear  as  if  the  three-level  factors 
were  in  fact  quantitative  and  equally  spaced.  Algorithm  I  systematizes  the 
calculation  of  the  factor  effects  for  the  2°  3m  series  of  designs. 

Algorithm  II  yields  the  set  of  fitted  values,  and  hence  the  residuals,  based 
on  those  factor  effects  which  have  been  judged  to  be  non-negligible.  The  two 
algorithms  have  additional  and  possibly  more  Important  uses  in  studying 
fractionated  2n3m  factorial  experiments.  Algorithm  I  can  be  used  to 
facilitate  the  writing  down  of  the  cross-product  matrix  for  any  desired  set  of 
factor  effects  for  a  specified  set  of  treatment  comparisons.  For  the  special 
case  of  the  standard  2^  P  series  of  designs,  the  two  algorithms  can  be  used 
to  find  the  set  of  defining  contrasts  corresponding  to  a  given  set  of 
treatment  combinations,  or  to  find  the  set  of  treatment  combinations 
corresponding  to  a  given  set  of  defining  contrasts.  That  the  analysis  breaks 
the  sources  of  the  three-level  factor  and  its  interactions  into  single- 
degrees-of-f reedom  components  suggests  that  the  data  might  be  inspected  using 
the  normal  or  half-normal  plot  (p.  249). 
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MacDonald,  P.  The  analysis  of  a  2  experiment  by  means  of  ranks.  Journal 

of  the  Royal  Statistical  Society.  Series  C  (Applied  Statistics),  1971,  20. 
259-270. 

A  method  is  proposed  for  the  analysis  of  variance  of  ranked  data  from  a 
factorial  experiment  of  size  2°  which  may  be  replicated  or  not.  A  test  of 
the  hypothesis  that  a  given  linear  contrast  of  means  is  zero  is  based  on  the 
corresponding  contrast  of  the  ranks  of  the  observations  in  each  replicate.  An 
appropriate  critical  region  for  the  test  is  based  on  the  rank  sum  test,  which 
is  equivalent  to  the  new  test  under  certain  null  hypotheses.  Two  other  forms 
of  the  test  are  also  suggested.  Estimates  of  the  population  contrasts  may  be 
obtained  by  means  of  an  ad  hoc  procedure  for  resolving  the  ambiguity  in  the 
means  of  the  test  statistic,  together  with  normalization.  The  method  is 
intended  especially  for  the  case  where  no  quantitative  observations  can  be 
made,  or  with  the  unreplicated  experiment  when  quantitative  measurements  are 
possible.  An  example  is  given  in  which  the  half-normal  plot  is  employed  to 
identify  the  significant  effects  (p.  263-269). 

GRAPHICAL  TECHNIQUES  IN  STATISTICAL  DATA  ANALYSIS 

Feder,  P.  I.  Graphical  techniques  in  statistical  data  analysis  —  tools  for 
extracting  information  from  data.  Technome tries .  1974,  16,  287-299. 

A  case  study  is  presented  that  demonstrates  the  usefulness  of  graphical 
techniques  in  the  analysis  of  data.  Crossplotting,  probability  plotting,  and 
graphical  multiple  comparison  procedures  are  discussed.  It  is  shown  how 
graphical  displays  both  motivate  the  use  of  and  help  interpret  the  results  of 
more  usual  numerical  techniques.  The  entire  discussion  is  centered  around  the 
analysis  of  a  particular  set  of  experimental  data  involving  a  comparison  of 
treatments.  It  is  shown  step  by  step  how  the  various  graphical  techniques, 
used  in  conjunction  with  more  classical  techniques,  extract  the  information 
contained  in  the  data. 


Nair,  V.  N.  On  the  behavior  of  some  estimators  from  probability  plots. 

Journal  of  the  American  Statistical  Association.  1984,  79*  823-831. 

Pitting  the  straight  line  through  a  plot  is  used  to  estimate  the  standard 
error  of  the  data.  The  author  investigates  the  properties  of  those 
estimators.  Estimators  from  weighted  least  square  lines  are  considered  and 
their, asymptotic,  finite-sample,  robustness,  and  optimality  properties  are 
discussed.  Included  among  these  are  the  ordinary  least  squares  estimators  and 
estimators  from  least  squares  lines  fitted  after  trimming  or  Vinsorizing  some 
of  the  extreme  order  statistics.  The  trimmed  least  squares  estimators,  with 
trimming  properties  reasonably  chosen,  provide  a  compromise  between  efficiency 
and  robustness. 

Wilk,  M.  B.,  and  Gnanadesikan,  R.  Probability  plotting  methods  for  the 

analysis  of  data.  Biometrlka,  1968,  55,  1-17. 

This  paper  describes  and  discusses  graphical  techniques,  based  on  the 
primitive  empirical  cumulative  distribution  function  (e.c.d.f.)  and  on 
quantile  (Q-Q)  plots,  percent  (P-P)  plots,  and  hybrids  of  these,  which  are 
useful  in  assessing  a  one-dimensional  sample,  either  from  original  data  or 
resulting  from  analysis.  Areas  of  application  include:  the  comparison  of 
samples;  the  comparison  of  distributions;  the  presentation  of  results  on 
sensitivities  of  statistical  methods;  the  analysis  of  results  on  sensitivities 
of  statistical  methods;  the  analysis  of  collections  of  contrasts  and  of 
collections  of  sample  variances;  the  assessment  of  multivariate  contrasts;  and 
the  structuring  of  analysis  of  variance  mean  squares.  Many  of  the  objectives 
and  techniques  are  illustrated  by  example.  Normal  and  half-normal  plots,  one 
class  of  six  defined  orthogonal  analysis-of-variance  situations,  are  discussed 
in  Section  6.2,  titled,  "The  univariate  single  degree  of  freedom  case." 


MULTIPLE  COMPARISON  PROCEDURES 

Gnanadesikan,  R.,  and  Lee,  E.  T.  Graphical  techniques  Cor  internal  compari¬ 
sons  amongst  equal  degree  of  freedom  groupings  in  multiresponse  experi¬ 
ments.  Biometrika.  1970,  57,  229-237. 

Probability  plotting  methods  are  given  for  two  sunmary  statistics  derived 
from  equal  degree-of-freedom  sum  of  product  matrices.  The  methods  are  useful 
for  graphical  internal  comparisons  of  the  'magnitudes*  of  the  sum  of  products 
matrices.  Possible  applications  with  multiresponse  data  include  the 
simultaneous  assessment  of  all  the  main  effects,  or  all  of  the  interactions  of 
the  same  order,  in  a  factorial  experiment  with  m  >  3  levels  for  each  factor, 
and  the  comparisons  of  several  observed  covariance  matrices,  for  example, 
within-group  covariance  matrices  in  a  multiresponse  analysis  of  variance  or 
discriminative  analysis,  each  based  on  the  same  number  of  replicate 
observations.  Illustrative  applications  are  included. 

Kurtz,  T.  E. ,  Link,  R.  F.,  Tukey,  J.  W. ,  and  Wallace,  D.  L.  Short-cut  multiple 
comparisons  for  balanced  single  and  double  classifications:  Part  I, 

Results.  Technometrics,  1965,  7,  95-161. 

This  paper  includes  a  general  and  considered  discussion  of  multiple 
comparison  procedures:  When  they  should  and  should  not  be  used,  the  importance 
of  confidence  procedures  and  their  advantages  over  significance  procedures, 
choice  among  multiple  comparisons  confidence  procedures,  and  choice  and 
description  of  error  rates.  It  does  not  attempt  to  compare  multiple 
comparison  and  multiple  decision  procedures.  The  authors  make  an  interesting 
distinction  among  experiments  intended  for  "decision,  significance, 
confidence,  or  selection." 

Following  the  Kjrtz  paper  in  the  same  Journal,  comments  on  it  were  made  by 
J.  E.  Jackson,  p.  163-165,  and  F.  J.  Anscombe,  p.  167-168;  Kurtz  et  al. 
replied  (p.  169)  to  Anscombe's  comments. 
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Andrews,  D.  P. ,  Gnanadesikan,  R.,  and  Varner,  J.  L.  Transformation  of  multi¬ 
variate  data.  Biometrics.  1971,  27..  825-840. 

Methods  which  are  extensions  of  Box  and  Cox's  work  (1964)  are  proposed  for 
obtaining  data-based  transformations  of  multivariate  observations  to  enhance 
the  normality  of  their  distribution  and  also  possibly  to  simplify  the  model 
(e.g.,  improve  additivity,  homoscedasticity,  etc.).  Specifically,  power 
transformations  of  the  original  variables  are  estimated  to  affect  both 
marginal  and  joint  normality.  A  method  for  improving  directional  normality  is 
also  described.  Examples  are  included  to  illustrate  some  properties  of  the 
method  with  normal  plots  being  employed  to  show  the  changes  that  occur  in  the 
data. 

Miller,  R.  G. ,  Jr.  Developments  in  multiple  comparisons,  1966-1976.  Journal 
of  the  American  Statistical  Association.  1977,  72,  779-788. 

A  bibliography  of  255  references  is  supplied  including  some  references  on 
graphical  techniques.  However,  papers  on  ranking  and  selection  were  omitted 
as  well  as  those  on  outlier  detection. 


SECTION  VIII 


CONCLUSIONS 

From  both  aggregate  and  individual  plots,  whether  quantified  or 
interpreted  by  eye,  whether  normal  or  half-normal,  our  analyses  clearly  show 
that  the  use  of  plots  cannot  be  relied  upon  as  a  certain  and  sole  means  of 
interpreting  the  results  from  a  2  factor ial-type  experiment.  They  have 
their  advantages  but  they  also  have  their  limitations,  and  cannot  be 
considered  the  final  word  as  a  detection  tool. 

If  we  do  not  aspire  to  quantify  our  decisions  regarding  the  probability 
that  a  particular  contrast  is  real,  the  complications  are  less  severe.  We  can 
look  at  the  plots,  and  sometimes  pick  out  most  of  the  real  effects. 

Sometimes,  we  won't  be  able  to,  but  in  all  cases,  it  is  the  marginal  real 
contrasts  that  will  be  the  most  difficult  to  detect  and  the  marginal  error 
contrasts  that  will  be  most  difficult  to  reject. 

The  nature  of  the  plots  work  against  us.  We  have  seen  how  the  E-  and  the 
R-spillovers  occur  most  frequently  and  over  more  ranks  when  the  size  of  the 
real  effects  are  smaller  (and  these  are  the  marginal  ones  that  need  the  most 
help  in  detecting),  when  the  number  of  real  effects  increase  (and  for 
screening  experiments  we  expect  more  rather  than  fewer  real  effects),  and  when 
the  size  of  the  experiment  is  smaller  (which  we  prefer  for  the  sake  of 
economy) . 

It  had  been  hoped  that  the  guardrails  would  help  reduce  these  problems. 

But  this  kind  of  quantification  has  proven  to  be  little  more  than  a  numbers 
game,  a  quasi-scientif ic  effort  that  still  requires  the  investigator  to  use 
the  same  Judgment  he  might  have  used  had  he  not  had  the  guardrails.  For 
example,  if  we  don't  pick  up  enough  real  effects  when  a  particular  guardrail 
at  one  probability  level  is  used,  we  are  advised  to  raise  the  probability 
level  until  we're  satisfied.  After  all,  the  guardrails  set  the  error  rate  for 
calling  an  error  contrast  real,  a  decision  which  we  are  less  concerned  with  in 
screening  experiments  than  in  calling  the  real  contrast  an  error. 


Calculating  guardrails  is  a  circular  exercise.  That  means  that  to 
discover  the  correct  answers,  one  must  first  know  the  correct  answers.  The 
guardrails  that  we  need  to  help  us  decide  which  contrasts  are  real  depend  on 
an  accurate  estimate  of  the  population  sigma,  inferred  from  one's  data. 
Calculating  the  slope  of  the  standardized  error  contrasts,  or  some  part  of 
them,  is  currently  the  best  way  to  make  this  estimate.  But  to  standardize  the 
contrasts,  one  needs  to  know  the  population  sigma,  which  is  what  we're  trying 
to  determine.  As  far  as  using  the  contrast  closest  to  the  0.683  probability 
level  for  the  half-normal  plot  (as  originally  suggested  by  Daniel)  or  0.34  and 
0.84  probability  levels  for  the  normal  plot  as  the  initial  estimate  of  sigma, 
that  can  only  lead  to  incorrect  estimates  unless  there  are  a  very  few  real 
effects,  a  condition  not  to  be  expected  in  screening  experiments.  We  have 
seen  how  much  a  wrong  guess  distorts  the  final  sigma  estimate. 

But  the  problem  of  circularity  doesn't  stop  there.  The  slope  that  will 
serve  as  the  best  estimate  of  the  population  sigma  is  the  plot  relating  the 
standardized  error  contrasts  to  the  estimated  values  of  normal  order 
statistics  for  the  correct  number  of  error  contrasts.  But  we  can  only  know 
the  correct  number  of  error  contrasts  if  we  know  the  correct  number  of  real 
contrasts,  the  purpose  of  this  entire  exercise  in  the  first  place. 

Guardrails  just  don’t  add  anything  that  might  not  be  achieved  using  an 
informed  rational  approach,  some  appreciation  of  the  E-  and  R-spillover  for 
the  particular  experimental  space,  and  a  liberal  attitude  toward  the  inclusion 
of  some  error  terms.  A  cursory  examination  of  the  literature  in  which  plots 
are  employed  seems  to  suggest  that  today  most  investigators  agree  with  this 
point;  at  least  those  who  use  probability  plots  don't  use  guardrails.  The 
ineffectiveness  of  guardrails  does  not  eliminate  the  usefulness  of  probability 
plots.  F-tests  have  many  of  the  same  weaknesses  in  practice  as  plots  have  and 
have  not  some  of  their  advantages.  The  author  still  believes  that  when  used 
judiciously  plots  are  still  the  best  way  to  interpret  the  results  from  a 
screening  experiment. 
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What  advantages  do  probability  plots  have  for  identifying  the  critical 
factors?  For  one  thing,  they  require  an  organization  of  the  data  in  a  way 
that  may  facilitate  interpretation.  By  ordering  the  contrasts,  one  can 
frequently  infer  from  the  larger  ones  which  marginal  ones  are  also  likely  to 
be  real.  If  the  proper  preliminary  analysis  is  done  before  the  experiment 
begins,  a  rational  approach  is  unlikely  to  be  any  worse  than  using  guardrails 
to  make  detection  decisions  and  may  be  a  whole  lot  better.  For  another  thing, 
plots  are  a  quick  way  to  examine  one's  data.  One  doesn't  need  a  computer, 
only  a  table  of  normal  order  statistics.  Then  too,  when  one  is  working  with 
unreplicated  designs,  the  plots  can  be  used  with  no  major  assumptions,  whereas 
if  one  still  insists  on  using  an  analysis  of  variance  for  significance 
testing,  it  would  be  necessary  to  dream  up  an  imaginary  error  variance, 
probably  based  on  some  combination  of  higher-order  interaction  effects,  which 
aren't  likely  to  be  available  in  the  early  stages  of  a  screening  experiment 
anyway.  As  we  move  through  the  sequential  stages  of  an  experiment,  adding 
blocks  to  unravel  confounded  effects,  the  effectiveness  of  plots  improve 
because  of  the  larger  n.  As  an  added  advantage,  probability  plotting  can  be 
another  way  to  examine  one's  data  for  abnormalities. 

But  should  we  use  normal  or  half-normal  plots?  There  were  some 
indications  that  the  normal  plot  had  advantages  in  terms  of  the  R-spillover 
over  the  half-normal,  but  not  much.  On  the  other  hand,  the  half-normal 
eliminates  the  need  to  check  both  ends  of  the  continuum  since  the  absolute 
effects  are  all  positive.  Of  course,  once  data  are  obtained,  they  can  be 
plotted  in  both  ways.  Inspection  of  both  plots  can't  hurt  and  may  provide 
clues  to  borderline  cases.  It  was  seen  that  if  one  combined  the  data  from  a 
normal  and  a  half-normal  plot  when  making  an  initial  estimate  of  sigma  for 
purposes  of  standardizing  the  data,  the  average  of  the  final  estimate  of  sigma 
from  both  gave  a  more  accurate  estimate  than  either  alone  because  of  the 
reciprocal  relationship. 

From  our  literature  search,  we  noted  that  investigators  continue  to  use 
the  plot  as  an  aid  to  interpreting  different  kinds  of  data  and  for  different 
purposes.  Some  continue  to  propose  alternative  solutions  to  problems  for 
which  probability  plots  were  originally  intended  to  solve. 


ADDITIONAL  EFFORTS 


Some  problems  that  might  be  of  interest  to  pursue  (although  to  do  so  may 
not  necessarily  be  cost-effective)  would  be  to  examine  the  effect  of 
confounding  on  the  effectiveness  of  probability  plots.  With  screening 
experiments,  main  effects  are  confounded  with  all  odd  higher-order  Interaction 
effects  and  two-factor  interaction  effects  are  confounded  not  only  with 
two-factor  interactions,  in  some  cases,  but  also  the  even  higher-order 
interaction  effects.  The  plot  only  treats  the  combination  as  a  single  degree 
of  freedom.  But  it  is  not  clear  whether  or  how  the  probability  of  detecting 
critical  factors  is  affected  when  four  effects  are  confounded  versus  eight 
effects  in  an  aliased  interaction  string  of  a  fractional  factorial. 

Another  thing  that  might  be  done  would  be  to  expand  the  relatively  simple 
relationship  discovered  between  r  and  d  and  the  R-spillover  and  include  n  and 
to  transform  the  data  so  as  to  improve  the  linearization  of  the  equation. 
[Note:  This  is  being  done.]  The  relation  between  R-spillover  (and 
E-spillover)  to  contrast  degradation  still  lends  itself  to  quantification 
provided  one  can  find  pragmatic  advantages  in  knowing  how  to  use  this 
information  to  interpret  the  individual  experiment.  Also,  finding  ways  to 
relate  results  when  the  sizes  of  multiple  real  effects  are  different  rather 
than  equal,  as  in  most  of  our  examples,  might  make  tables  of  R-spillover  have 
greater  utility. 

Finally,  it  would  be  useful  to  find  what  has  and  hasn't  been  done 
regarding  the  use  of  probability  plots  to  (1)  examine  the  quality  of  one's 
data,  and  (2)  interpret  the  results  from  multifactor-multivariate  experiments. 
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APPENDIX  A 


LITERATURE  SEARCH 

A  search  was  made  in  the  Science  Citation  Indexes  from  1955  to  1986  and  in 
the  CompuMath  Citation  Indexes  from  1981  to  1986  for  any  references  to 
Daniel’s  (1959)  and  Zahn’s  (1975a,b)  papers  on  half-normal  plots. 

The  list  below  provides  the  authors,  journals,  and  location  within  the 
journals  of  articles  which  referred  to  Daniel's  and  Zahn's  papers.  None  of 
the  report  titles  are  given,  although  they  may  be  found,  if  desired,  in  the 
Sources  Indexes  which  accompany  the  Science  Citation  and  CompuMath  Citation 
Indexes.  Even  in  its  abbreviated  form,  however,  this  list  provides  enough 
information  for  a  user  to  go  directly  to  the  appropriate  journals  to  find  an 
article. 

The  list  is  useful  for  those  who  wish  to  see  how  normal  and  half-normal 
plots  have  been  used.  A  majority  of  the  articles  listed  below  describe 
applications  rather  than  analyze  or  otherwise  discuss  or  examine  the  operating 
characteristics  of  these  plots.  It  cannot  be  assumed  that  this  list  is 
complete;  for  example,  it  does  not  include  any  references  from  government  or 
industrial  publications.  The  list  does  indicate  the  impact  Daniel's  paper  has 
had  on  the  interpretation  of  experimental  data.  That  fact  is  particularly 
interesting  in  the  light  of  Zahn's  criticisms  and  the  fact  that  Zahn’s  paper 
—  undoubtedly  more  technically  correct  than  Daniel's  —  is  only  referenced  in 
these  same  volumes  a  few  times.  All  of  the  listed  papers  were  found  in  the 
Science  Citation  Index  except  those  marked  with  an  #,  which  came  from  the 
CompuMath  Citation  Index. 

The  references  are  organized  according  to  the  years  contained  in  the 


individual  volumes  of  these  Indexes. 


ARTICLES  REFERENCING  DANIEL’S  (1959)  PAPER  ON  HALF-NORMAL  PLOTS 


Index  Vol.  Year(s)  1st  Author 
1955-1969  * 

Cox,  DR 
Draper,  NR 
Fedorov,  VD 
Flenberg,  SE 
Gorman,  JV 
Govindar,  VM 
Healy,  MJR 
Hills,  M 
Holms,  AG 
Kurtz,  TE 
Margolin,  BH 
Myers,  MH 
Shapiro,  SS 
Stowe,  RA 
Stowe,  RA 
Webb ,  SR 
Wilk,  MB 


1955-1964 


1970-1974 


Addelman,  S 
Addelman,  S 
Banerlee,  KS 
Box,  GEP 
Draper,  NR 
Eland t,  RC 
Hunter,  JS 
Leone ,  FC 
Stevens,  CD 
Sweeny,  R 
Truax,  HM 
Tukey,  jw 
Wilk,  MB 
Wilk,  MB 
Wilk,  MB 


Abdulrah,  YA 
Andrews ,  DF 
Andrews,  DF 
Anscombe ,  FJ 
Congdon,  CC 
Dyer,  DD 
Egorov,  NS 
Evans,  DA 
Feder,  PI 
Fleming,  AF 
Gnanades,  R 
Gnanades,  R 


Journal 

Volume 

1st  Pane 

Year 

Technomet 

9 

481 

67 

Can  Math  B 

11 

475 

68 

Dan  SSSR 

188 

913 

69 

Appl  Stat 

18 

153 

69 

Technomet 

8 

27 

66 

Ind  Eng  PDD 

7 

573 

68 

Appl  Stat 

17 

157 

68 

Blometrika 

56 

249 

69 

Technomet 

11 

725 

69 

Technomet 

7 

95 

65 

Technomet 

9 

245 

67 

J  Chron  Dis 

19 

923 

66 

Blometrika 

52 

591 

65 

Ind  Eng  Ch 

58 

36 

66 

Ind  Eng  Ch 

61 

11 

69 

Technomet 

10 

535 

68 

Blometrika 

55 

1 

68 

Technomet 

4 

21 

62 

Technomet 

6 

365 

64 

Ann  Math  St 

34 

1068 

63 

Technomet 

4 

301 

62 

Biometrics 

20 

443 

64 

Technomet 

3 

551 

61 

Technomet 

6 

41 

64 

Technomet 

3 

543 

61 

J  Pharm  Exp 

141 

267 

63 

Ind  Eng  Ch 

53 

329 

61 

J  Am  Oil  Ch 

37 

650 

60 

Ann  Math  St 

33 

812 

62 

Ann  Math  St 

36 

613 

64 

J  Am  Stat  A 

58 

152 

63 

Technomet 

4 

1 

67 

Chem  Eng  Sc 

28 

1273 

73 

Biometrics 

27 

825 

71 

J  Roy  Sta  C 

22 

192 

73 

Am  Statistn 

27 

17 

73 

J  Nat  Cane 

45 

1055 

70 

Technomet 

15 

489 

73 

Mikrobiolog 

42 

863 

73 

J  Roy  Sta  A 

136 

153 

73 

Technomet 

16 

287 

74 

Med  J  Aust 

2 

429 

74 

Ann  Math  St 

41 

292 

70 

Biomet  r ika 

57 

229 

70 

1970-1974  (cont'd) 


1975-1979 


1st  Author 

Journal 

Gnanades,  R 

J  Roy  Sta  B 

Goldsmith,  PL 

J  Roy  Sta  C 

Goswami,  BC 

Text  Res  J 

Gut  tman ,  I 

Technomet 

MacDonal,  P 
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J  Roy  Sta  A 
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Coldwell,  RL 
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Microbiolog 
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Shimalla,  CJ 

Text  Res  J 
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Turnbull,  BW 

Biometrics 
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Zahn,  DA 

Technomet 

Zahn,  DA 

Technomet 

Volume  1st  Page  Year 


1st  Author 

Journal 

Volume 

1st  Paue 

Year 

Gusev,  NV 

Microbiolog 

49 

19 

80 

Holms,  AG 

Comm  Stat  B 

9 

51 

80 

Lam,  CL 

J  Food  Sci 

45 

1720 

80 

Maksimov,  VN 

Microbiolog 

49 

186 

80 

Archer,  RH 
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12 

46 

81 

Darby,  SC 

J  Roy  Sta  A 
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296 

81 

Eld in,  SH 

J  Appl  Poly  M 

26 

1431 

81 

Jurgenson,  IA 

Parazitolog 

15 

38 

81 

Bradu,  D 

Technomet 

24 

103 

82 

Schweder,  T 

Biometrika 

69 

493 

82 

#  Stirling.  VD 

Statistica 

31 

211 

82 

Wilkinson,  SA 

J  Food  Sci 

47 

844 

82 

Cardone ,  MJ 

J  Aoac 

66 

1257 

83 

Carroll,  MB 
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37 

31 

83 

Cochrane,  RL 

Biol  Reprod 

28 

134 

83 

Huck,  PM 

Water  Res 

17 

1403 

83 

Malajczu,  N 

Ann  Ap  Biol 

103 

57 

83 

Snee,  RD 

J  Qual  Tech 

15 

76 

83 

Cox ,  DR 

Int  Stat  R  R 

52 

1 

84 

Kotze,  TJV 

Appl  Stat 

33 

215 

84 

#  Mauro,  CA 

Manag  Sci 

30 

209 

84 

Nair,  VN 

J  Am  Stat  A 
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823 

84 

Ziegel,  ER 

Technomet  D 

26 

98 

84 

Funk,  W 
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128 

45 

85 

Kaitala,  S 

Ecol  Bull 
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84 

Prew,  RD 

J  Agr  Sci 

104 

135 

85 

Box,  GEP 

Technomet 

28 

11 

86 
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Index  Vol.  Year(s) 

1st  Author 

1975-1979 

Miller,  RG 

J  Am  Stat  A 

Sachs,  L 

Kiln  Woch 

1980 

Dixon,  WJ 

Ann  R  Pharm 

1981-1986 

None  listed 

Journ 


72 

55 
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APPENDIX  B 

EXPECTED  VALUES  OP  NORMAL  AND  HALF-NORMAL  ORDER  STATISTICS 
FOR  k  =■  31,  63,  and  127 

NORMAL  ORDER  STATISTICS 


ORDER 

statistic 

-2.05647 

-1.63167 

-1.38269 

-1.19804 


ORDER 
STATISTIC 
-2. 33781 
-1.95626 
-1.73  90  5 
-1.58179 
-1.45605 


6 

-0. 91688 

6 

-1.34981 

7 

-0.80066 

7 

-1.25699 

8 

-0.69438 

8 

-1.17388 

9 

-0. 59546 

9 

-1.09820 

10 

-0. 50206 

10 

-1.02834 

11 

-0. 41287 

11 

-0.96317 

12 

-0.32686 

12 

-0.90188 

13 

-0.24322 

13 

-0.84380 

14 

-0.16126 

14 

-0.78844 

15 

-0.08037 

15 

-0.73539 

16 

0.  00000 

16 

-0.68436 

17 

0.08037 

17 

-0.63504 

18 

0.16126 

18 

-0.58724 

19 

0.24322 

19 

-0.54073 

20 

0. 32686 

20 

-0.49537 

21 

0.41287 

21 

-0.45101 

22 

0.50206 

22 

-0.407  3 

23 

0. 59545 

23 

-0.36480 

24 

0.69438 

24 

-0.32273 

25 

0. 80065 

25 

-0.28122 

26 

0. 91689 

26 

-0.24019 

27 

1.04703 

27 

-0.19957 

28 

1.19804 

28 

-0.15927 

29 

1.38269 

29 

-0.11923 

30 

1.63167 

30 

-0.07938 

31 

2.05647 

31 

-0.03966 

32 

0.00000 

33 

0.03966 

34 

0.07938 

35 

0.11923 

36 

0.15927 

37 

0.19957 

38 

0.24019 

39 

0.28122 

40 

0.32273 

41 

0.36480 

42 

0.40753 

43 

0. 45101 

44 

0.49536 

45 

0. 54073 

46 

0.58724 

47 

0.63504 

48 

0.68436 

49 

0.73539 

50 

0.78844 

51 

0. 84380 

52 

0. 90188 

53 

0. 96317 

54 

1.02833 

55 

1.09820 

56 

1.17388 

57 

1.  25698 

58 

1.34981 

59 

1.45605 

60 

1.S8178 

61 

1. 73905 

62 

1.95626 

63 

2.33781 

ORDER 

I  STATISTIC 

1  -2.59185 

2  -2.24250 

3  -2.04751 

4  -1.90849 

5  -1.79870 

6  -1.70701 

7  -1.62779 

8  -1.55772 

9  -1.49450 

10  -1.43669 

11  -1.38338 

12  -1.33376 

13  -1.28721 

14  -1.24331 

15  -1.20171 

16  -1.16204 

17  -1.12415 

18  -1.08776 

19  -1.05286 

20  -1.01913 

21  -0.98650 

22  -0.95496 

23  -0.92428 

24  -0.89449 

25  -0.86544 

26  -0.83714 

27  -0.80947 

28  -0.78239 

29  -0.75589 

30  -0.72996 

31  -0.70447 

32  -0.67941 

33  -0.65478 

34  -0.63058 

35  -0.60669 

36  -0.58317 

37  -0.55997 

38  -0.53704 

39  -0.51443 

40  -0.49207 

41  -0.46995 

42  -0.44805 

43  -0.42635 

44  -0.40488 

45  -0.38359 

46  -0.36245 

47  -0.34149 

48  -0.32068 

49  -0.30000 

50  -0.27947 

51  -0.25902 

52  -0.23869 

53  -0.21847 

54  -0.19833 

55  -0.17828 

56  -0.15829 

57  -0.13837 

58  -0.11850 

59  -0.09868 

60  -0.07889 

61  -0.0S914 

62  -0. 03942 

63  -0.01971 

64  0.00000 

•*.  “V  **.  “'-  ■V 
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0.01971 

0.03942 

0.05914 

0.07889 

0.09868 

0.11850 

0. 13837 

0.15829 

0. 17828 

0.19833 

0.21847 

0.23870 

0. 25903 

0.27947 

0.30000 

0.32068 

0. 34150 

0.36245 

0.38360 

0.40489 

0.42636 

0.44805 

0.46994 

0.49208 

0.51443 

0.53704 

0. 55997 

0.58317 

0.60670 

0.63058 

0.65479 

0.67941 

0.70447 

0.72997 

0.75589 

0.78239 

0.80948 

0.83714 

0.86544 

0.89451 

0.92427 

0.95497 

0.98651 

1.01915 

1.05286 

1.08776 

1. 12413 

1.16205 

1.20171 

1.24333 

1. 28722 

1.33375 

1 . 38340 

1.43670 

1. 49449 

1.55775 

1.62778 

1.70698 

1. 79873 

1. 90851 

2.04752 

2.24250 

2.  59185 
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APPENDIX  B  (cont'd) 
HALF-NORMAL  ORDER  STATISTICS 


Number 

Ordar  Statistic 

Ntsmbar 

Orrfar  Statist 

1 

0. 037755 

1 

0. 019230 

2 

0. 077344 

C> 

0. 039096 

3 

0. 116146 

3 

0. 058495 

4 

0.  155737 

4 

0. 077851 

5 

0. 196288 

,  5 

0. 097564 

6 

0. 236848 

6 

0. 117018 

7 

0. 277176 

7 

0. 137275 

8 

0. 318803 

8 

0. 156933 

9 

0. 361311 

9 

0.  176493 

10 

0.  402889 

10 

0. 196258 

t  1 

0.  446393 

11 

0. 216200 

12 

0. 490765 

12 

0. 235971 

13 

0. 535489 

13 

0. 256196 

14 

0. 580973 

14 

0. 276680 

IS 

0. 629700 

15 

0. 297813 

16 

0. 679548 

16 

0. 318330 

17 

0. 732111 

17 

0.  339698 

IS 

0.  786062 

18 

0. 360498 

19 

0. 842622 

19 

0. 381368 

20 

O. 901025 

20 

0. 402650 

21 

0.  961400 

21 

0. 423443 

22 

1.  026500 

22 

0. 445411 

23 

1. 096640 

23 

0.  467422 

24 

1.  173480 

24 

0. 489720 

23 

1.  236180 

25 

O. 512424 

26 

1.  348710 

26 

O. 534829 

27 

1.  4376S0 

27 

O. 557291 

23 

1.  584930 

28 

0. 579992 

29 

1.  744760 

29 

0. 603416 

30 

1.  959020 

30 

O.  627738 

31 

2.  342680 

31 

0. 651868 

32 

0.  67639B 

33 

0.  701125 

34 

0.  727618 

35 

0.  753351 

36 

0.  779807 

37 

0.  807111 

38 

0.  834626 

39 

0.  862614 

40 

0.  891383 

41 

0.  921166 

42 

0.  952432 

43 

0.  983861 

44 

1.  016860 

43 

1.  050560 

46 

1.  085090 

47 

1.  121530 

48 

1.  159500 

49 

1.  199540 

50 

1.  241350 

51 

1.  285290 

^C? 

1. 332120 

53 

1.  383010 

54 

1.  435900 

55 

1  493260 

5cj 

1 .  555400 

57 

1 . 624930 

58 

1  .  704560 

59 

1  .  797090 

to 

1  908770 

61 

2  047880 

62 

2.  240550 

63 

2.  592380 
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APPENDIX  B  (cont’d) 


HALF-NORMAL  ORDER  STATISTICS 


Number 

Order  Statis' 

1 

0. 009B09 

2 

0. 019836 

3 

0. 029776 

4 

0. 039751 

5 

0. 049482 

6 

0. 059341 

7 

0. 069374 

e 

0. 079189 

9 

0. 089033 

10 

0. 098757 

ii 

0. 108474 

12 

0.  1 18526 

13 

0. 128607 

14 

0.  138253 

15 

0.  148170 

16 

0.  158008 

1  7 

O.  167908 

ie 

0.  177665 

19 

0.  187725 

20 

0.  197852 

o  •» 

G.  4 

0. 207862 

2  2 

0. 217746 

23 

O. 227816 

24 

0. 237906 

25 

O. 247963 

26 

O.  257876 

27 

0. 268187 

28 

0. 278347 

29 

O. 288580 

30 

0.  298931 

31 

0. 309051 

32 

0. 319467 

33 

0. 329850 

34 

0. 340470 

35 

0. 350732 

36 

0. 361337 

37 

0. 371741 

38 

0. 382275 

39 

0.  392913 

40 

0. 403483 

41 

0. 414017 

42 

0. 424991 

43 

0. 436135 

44 

0.  446763 

45 

0. 457691 

46 

0. 468713 

47 

0. 479703 

48 

0. 490531 

49 

0. 501775 

50 

0. 512986 

51 

0. 524642 

52 

0  535977 

0.  547383 

54 

0.  558796 

55 

0.  570297 

56 

0.  50  1  69 7 

57 

0  593361 

58 

0. 604956 

59 

0. 616701 

60 

0. 628569 

61 

0. 640321 

62 

0. 652393 

63 

0. 664481 

64 

0. 676715 

65 

0.  689347 

66 

0.  701778 

67 

0. 714480 

68 

0.  727106 

69 

0.  740169 

70 

0.  753378 

71 

0.  766354 

72 

O.  779366 

73 

O.  792861 

74 

0.  806778 

75 

O.  B20474 

76 

0.  834340 

77 

0.  848549 

78 

0.  862394 

79 

O.  876830 

80 

0.  891408 

81 

0.  905992 

82 

0. 920773 

e3 

0. 935736 

84 

0. 951 191 

e5 

0. 967161 

86 

0. 983027 

87 

0. 998870 

ee 

1. 014870 

89 

1. 031260 

90 

1. 048090 

91 

1.  065140 

92 

1. 083030 

93 

1. 100680 

94 

1. 119210 

95 

l.  137670 

96 

1.  156790 

97 

1.  176400 

98 

1.  195660 

99 

I.  215710 

100 

1.  236340 

101 

1. 257680 

102 

1. 279600 

103 

1. 302500 

104 

1. 325520 

105 

I . 349420 

106 

1. 373820 

107 

l. 399130 

108 

1 . 425780 

109 

1  453400 

1  1  0 

1  482160 

1  1  1 

1  512110 

1  12 

l  543300 

1  1  3 

1  576030 

1  1  4 

1  .  612090 

1  1  5 

1  .  649600 

1  16 

1  689-160 

118 

l.  776400 

119 

1.  826620 

120 

1. 881770 

121 

1. ^43290 

122 

2. 012240 

123 

2. 094940 

124 

2. 193610 

125 

2. 318260 

126 

2. 497170 

127 

2. 828630 

APPENDIX  C 

PROGRAM  FOR  GENERATING  DATA  BASES  AND  FOR  PERFORMING 
SUBSEQUENT  ANALYSES  USED  IN  THIS  REPORT* 

f  f-p 

This  program  constructs  experimental  designs  o f  the  2  (or  2  )  type 

and  will  produce  summary  or  "estimated"  expected  results  for  any  combination 
of  real  and  nonreal  effects  with  minor  modifications  by  the  user-  The  program 
also  produces  a  variety  of  other  statistics  and  output  which  can  be  used  to 
investigate  the  operating  characteristics  of  raultifactor  experiments  and 
normal  plots.  The  program  works  by  first  constructing  an  experimental  design 
of  the  2^  type  with  the  matrix  filled  with  -Is  and  +ls  to  represent  the  (2) 
levels  of  the  factors.  Then  random  normal  values  (N(0,1))  are  generated  and 
"assigned"  as  data  points  to  the  experiment.  There  are  options  for  assigning 

more  than  one  data  point  to  each  cell  of  the  design  matrix. 

The  effects  or  contrasts  for  the  experiment  are  then  computed.  Real  or 

"true*  effect  values  are  then  added  to  the  estimated  nonreal  contrasts  and  the 
effects  are  then  sorted  according  to  size.  This  process  is  repeated  some 
number  of  times  (until  desired  accuracy  is  achieved  —  5000  in  this  report), 
and  summary  statistics  giving  means  and  standard  deviations  for  the  sorted 
contrasts  are  computed.  The  program  also  has  subroutines  for  computing 
standard  normal  order  statistics  for  any  number  of  ordered  values  and  these 
can  be  printed  for  both  real  and  nonreal  effects  for  comparison  purposes.  In 
its  current  form,  certain  selected  summary  statistics  are  printed  to  a  disc 
file  which  can  then  be  subjected  to  further  analysis  using  other  statistical 
packages.  In  this  report,  the  normal  plot  routines  in  the  BMDP  statistical 
package  were  used  to  plot  the  obtained  ordered  values.  This  method  was  used 
to  obtain  normal  plots  for  the  outcome  from  a  single  experiment,  the  ordered 
summary  statistics  for  all  results,  and  the  ordered  summary  statistics  for 
intermediate  results.  The  user  can  modify  this  output  as  necessary  to  conduct 
supplemental  analyses.  Currently,  the  program  will  also  calculate  slope 
statistics  for  various  portions  of  the  normal  plot. 


*  Questions  regarding  this  program  should  be  addressed  to  Dr.  Daniel  P. 
Westra,  Essex  Corporation,  1040  Woodcock  Road,  Orlando,  FL  32813. 


* 
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user  must  input  the  following  to  the  program 


c 

The  size  of  the  experimental  design  (expressed  as  2C). 
The  number  of  real  effects. 

The  magnitude  of  each  real  effect. 

Number  of  data  points  per  cell  (default  to  one). 


c 

c 

c 

c 

c 

c 

c 

c 

c 

c 

c 

c 

c 

c 

c 

c 

c 

c 

c 

c 

c 

c 

c 

c 

c 

c 

c 

c 

c 

c 

c 

c 

c 

c 

c 

c 

c 

c 

c 

c 

c 

c 

c 

c 

c 

c 


c 


THIS  PROGRAM  WAS  WRITTEN  ON  A  VAX-780  USING 
VAX  FORTRAN-77*.  THE  FILE  NAME  WES. FOR  . 

TO  COMPILE:  FOR  WES 

TO  TASK  BUILD:  LINK  WES 


INPUTS:  THE  INPUTS  TO  THIS  PROGRAM  ARE 

1)  THE  SIZE  OF  THE  DESIGN  MATRIX. 

2)  THE  NUMBER  OP  REAL  EFFECTS. 

3}  THE  MAGNITUDE  OF  EACH  REAL  EFFECT. 

OUTPUT:  THERE  ARE  FIVE  FILES  THAT  ARE  CREATED 

1)  SIMON.DAT  -  THIS  PILE  HAS  SUMMARY  DATA  AFTER 

EACH  100  RUNS. 

2)  WES. DAT  -  THIS  FILE  IS  THE  FINAL  SUMMARY  AFTER 

5000  RUNS . 

3)  SLOPE. DAT  -  THIS  FILE  WAS  CREATED  TO  BE  AN 

INPUT  FILE  TO  THE  BMDP  PACKAGE 
5D  TO  CALCULATE  THE  PLOT  OF  THREE 
SIGMA  METHODS. 

4)  SIMONBMD.DAT  -  THIS  FILE  WAS  CREATED  TO  BE  AN  INPUT 

FILE  TO  THE  BMDP  PACKAGE  5D  TO  PLOT 
NEWE  AFTER  EACH  100TH  RUN. 

5)  ZMD.DAT  -  THIS  FILE  WAS  CREATED  TO  BE  AN  INPUT 

FILE  TO  THE  BMDP  PACKAGE  5D  TO  PLOT  NEWE 
AFTER  5000  RUNS. 


IMPLICIT  REALM  (M) 

DIMENSION  D(128) ,E(127) ,P (1 20, 127) , EN (127) , 

X  NS  (126) , EN1 (127) , El  (127) , YEN (127) , YEN2 (127) 

DIMENSION  ESS (127) ,RN(128) f MAG (32) ,YENA(127) ,YENA2(127) 

INTEGER  P, COUNT (127), T30 , T50 , T70 , F30 , L30 , F50 , L50 , F7 0 , L70 , 

X  EFF (127) , POSEFF, NEGEFF,K1,K2, NEWK, P30E, J30E, P50E* J  50E 


rr 


REALM  Newell 27)  ,D30,D50,  D7i,NEWD,C0Ll (127) ,COL2<127) 

REAL *4  ONE (30) ,TWO(29) , THREE (29 ) , FOUR (27) ,riVE (26) 

REALM  SU< 25)  f  SEVEN  (24)  ,NEWYCN  (127)  , MEWYEN2  (1 27 > 

REALM  ZMOZN  (127)  ,  ZMDZE  ( 127)  ,  ZMOZE1  (127) , ZEIGHT (127) 

RCAL*4  Z EIGHT 1 (127) ,  HAT  (31 ,  5i ) 

C 

LOGICAL  TEST (127) 

C 

C  SIGN  MATRIX  USED  AS  BASIS  rOR  GENERATING  OTHER  SIZE  FACTORIALS. 
C 

DATA  ((P(I,0> ,J-l,3> ,1-1,4) 

X  -l.,l.,l,,l./ 

c 

OPEN (UNIT-11,  TYPE-* NEW* , NAME- • S LOPE . DAT. • ) 

C 

NR-S000 

SEEO-SECNOS (0.0) 

C 

C  UMAX  MUST  BE  INPUT  BY  USER  .  IT  SPECIFIES  SUE  OF  DESIGN 
C  ACCORDING  TO  2*#NMAX. 

C  NSCELL  ALSO  USER  INPUT.  GIVES  NO.  SUBS  PER  CELL. 

C 

WRITE (5,2) 

2  FORMAT (‘  ENTER  SIZE  Of  DESIGN  MATRIX  *) 

ACCEPT  3, NMAX 

3  PORMAT (12) 

NCKAX-NMAX -1 
NPCON-2  **NMAX 
NFEFF-NFCON-1 
NSCELL-1 

C 

IF  (NMAX  . EQ.  5)  THEN 
P30  -  12 
L30  -  20 
D30  -  9. 

F  50  -  9 
LS0  -  23 
D50  -  15. 

P70  -  6 
L70  -  26 
D70  -  21. 

NEWD  -  0.3535533 
ELSE  IP  (NMAX  .EQ.  6)  THEN 
F 30  -  23 
L30  -  41 
D30  -  19. 

F50  -  17 
L 50  -  47 
D50  -  31. 

F70  -  10 
L70  -  54 
D70  -  45. 


MCWD  -  I.2S 
END  IF 
C 

NTEM P-NFCON  *NSCELL 
C 

c  GENERATE  THE  DESIGN  MATRIX 
C 

DO  101  Nl-2, NCMAX 
NCI-2*  *N 1 
NC2-NC1*2 
NST-NC1 *1 
NEl-NCl-l 
NE2-NC2-1 
C 

c  DUPLICATE  EXISTING  MATRIX 
C 

DO  11  t -NST , NC  2 
DO  11  J - 1 , HE  1 

P(I,J)-P(I-NC1,J) 

11  CONTINUE 

C 

C  GENERATE  NEXT  FACTOR 
C 

DO  15  1-1, NCI 
P  (l, NCI)—  1. 

15  CONTINUE 

DO  16  I »NST, NC  2 
P  {  I  ,  NC  1  )  •  1  . 

16  CONTINUE 
C 

C  *ULT  TERNS  BY  NEW  FACTOR 
C 

DO  31  I-1,NC2 
DO  U  J -NST , NE  2 

P(I,J)-P(I ,NC1 ) *P (I , J-NC1 ) 
31  CONTINUE 

101  CONTINUE 
C 
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DO  20  I-1,NTEMP 
NS(l)-I 

29  CONTINUE 

DO  11  J-l.NPCFP 

ess  (j)>*. 

ei(j>  •  «. 

COLl(J)-0. 

ZMDZE(J)-0. 

ZMDZE1  (J)-0, 

ZHDZN(J)-0. 

COL2(J)-0. 

E<J).0. 

21  CONTINUE 

SLOPE30  •  9. 

SLOPE  50  -  «. 


SLOPE 70  -  I. 

T3«  -  I 
TS0  -  9 
T7»  -  9 
K1  -  0 
K2  «  0 
C 

DO  SI  1-1,32 

MAC  (I)  -  0.0 
91  CONTINUE 
WRITE (5,4) 

*  FORMAT  ( •  ENTER  NUMBER  OP  REAL  EFFECTS  •) 

ACCEPT  3,F 
DO  I  -  1,F 
TTPE-, I 
WRITE (S, 5) 

5  FORMAT  (•  ENTER  MACN1TUDE  OF  EFFECTS  •) 

ACCEPT  6, MAC (I) 

6  FORMAT (F7. i) 

I F ( MAC ( I )  .GT.  0.  ) P0SEFF-P0SEFF+1 
I F ( MAG ( I )  .LT.  0 . ) NEGEPF-NECEFF+1 
END  DO 
C 

C  THE  FOLLOWING  IF  BLOCK  IS  USED  TO  ASSICN  VARIABLES 
C  THE  CORRECT  VALUE  DEPENDING  OF  THE  THE  DESICN  SIZE 
C  AND  IF  A  30  OR  50  PERCENT  SLOPE  ABOUT  THE  CENTER 
C  IS  NEEDED. 

c 

IF  (NMAX  .EQ.  S)  THEN 
IF  (F  .LE.  10)  THEN 
F30E  -  8 
J30E  -  IS 
P50E  -  5 
J50E  -  19 

ELSE  IF  (F  , EQ.  12)  THEN 
P30E  -  6 
J30E  -  14 
PS0E  -  3 
J50E  >  17 

ELSE  IF  (F  .EQ.  IS)  THEN 
P30E  -  4 
J30E  -  12 
PS0E  -  1 
JS0E  -  IS 
END  IF 

ELSE  IF  (NMAX  .EQ.S)  THEN 
It  (F  .EQ.  •)  THEN 
P30E  -  19 
J30E  -  37 
P50E  -  13 
J 50E  •  43  ’ 

ELSE  IF  (F  .EQ.  12)  THEN 


P30E  -  17 
J30E  -  35 
P50E  -  11 
J50E  -  41 

ELSE  IF  (F  . EQ. 16)  THEN 


P30E 

-  15 

J3eE 

-  33 

PS0E 

-  9 

J52E 

-  39 

END  IP 

END  IP 
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THIS  WILL  ECHO  CHECK  THE  POSITIVE  AND  NEGATIVE 
VALUES  OP  THE  REAL  EFFECTS. 

TYPE*  t POSEPP 
TYPE* , NEGEFF 


Kl  -  NFEFF-POSEFF 
K2  -  NPEPP-P 

ORDER  STAT.  FOR  REAL  AND  NON-REAL  EFFECTS 

DO  I  -  I , NFEFF 

CALL  SCOR (I, NFEFF, SN# VAR) 

YEN  (I )  -  SN 
YEN2(I)“  VAR**. 5 

NEWYEN (I )  -  SN  | ORDER  STAT.  FOR  NFEFF 

NEWYEN2 (I )  -  VAR  *  * • $  I  STD. DEV.  FOR  ORDER  STAT. 

END  DO 

ORDER  STAT.  FOR  NON-REAL  EFFECTS 
DO  1-1 , K2 

CALL  SCOR (I,K2,SN,VAR) 

YENA(I)  -  SN 
YENA2  (I )  -  VAR**. 5 
END  DO 

ORDER  STAT.  FOR  REAL  EFFECTS 
DO  1-1, P 

CALL  SCOR (I ,  P, SN, VAR ) 

Z EIGHT (I )  -  SN 
ZEIGHTl(I)-  VAR**. 5 
END  DO 

CALL  SORT(YEN, NFEFF) 

CALL  SORT (YENA, K2) 

CALL  SORT(NEWYEN, NFEFF) 

CALL  SORT ( ZEIGHT, F) 

IF (NEGEFF  . EQ.  0)GOTO  9 


DO  Iil, NEGEFF  1LOOP 

EFF (I )  -  F 

YEN(I)  -  Z EIGHT  (I )  ♦  (MAG(I)/NEWD) 
YEN2  (I )  »  ZEIGHTl(I) 

END  DO 

IF  (NEGEFF  .GT.  0 )  NEGEFF  “NEGEFF -fl 


I LOOP  TWO 


IF  (NEGEFF  .EQ.  0)NEGEFFml 

DO  I-NEGEPF, Kl 
EFF  (I )  -  K2 

YEN (I)  -  YENA (I- (NEGEPF-1 ) ) 
YEN2  (I )  •  YENA2  (I«*  (NEGEFF-1)  ) 
END  DO 


Kl  -  Kl  ♦  1 
DO  I -Kl, NFEFF 
EFF(I)  -  F 

YEN (I)  -  ZEIGHT  (I-K2)  ♦  (MAG (I-K2) /NEWD) 
YEN 2 (I ) *  ZEICHT1 (I-K2) 

END  DO 


START  OF  LOOP  FOR  THE  10K  (OR  WHATEVER)  RUNS 


ITEMP-100 

NEWK-1 

DO  100  KK “1 , NR 

RANDOM  DRAW  FROM  N(0,1) 


CALL  RNORM01  (NTEMP, RN, SEED) 
DO  509  I -1,  NTEMP 

D ( I )-RN  (I  ) 

CONTINUE 


via 


s 


RANDOM  ASSIGNMENT  Of  •SUBJECTS*  TO  CONDITIONS 

NTEH2-NTEMP-1 
DO  30  I-l,NTEM2 
I P -NT  EM P- 1*1 
POS-IP 

RAN  -  MTH$RANDOH (SEED) 


30 

C 

c 

c 


43 


40 

C 


ICH-RAN*PQS+1 
IS-NS (IP) 

NS (IP)-NS (ICM) 

NS (ICM)-IS 
CONTINUE 

NUMBERS  REPRESENTING  SUBJECTS  MEAN  LEVEL  INPUT  AND  TALLIED 

DO  40  J-1,NPEFF 
K-l 

DO  40  I-l,NFCON 
C-0. 

PROVISION  TO  PUT  IN  MORE  THAN  1  SUB  PER  CELL. 

K2-K+NSCELL-1 
DO  43  Ki-K,K2 
1 1  "NS  (K  1 ) 

C-C+D (II ) 

CONTINUE 

C-C/NSCELL 

K-K2+1 

EN(J)-EN(J)+P(I,J)*C 

CONTINUE 

JJ2-NPCON/2 


.  .N 

& 

E 

y\ 

N 


c 

USE  ABS  VALUES  OP  MEAN  DIFFERENCES  HERE  IF  DESIRED. 

\v 

c 

■s 

DO  41  I-l.NFEFF 

- 

EN  (I )-  (EN  (I )  ) /FLOAT  (JJ2  J 

41 

CONTINUE 

! 

DO  K  -  1,F 

EN  (K  J  -  EN  (K )  +  MAG(K) 

END  DO 

DO  I  -  l.NFEFF 

EN1  (I)  .  EN  (I ) 

«/ 

END  DO 

*  * 

c 

c 

SORT  EFFECT  SIZES 

* 

c 

CALL  SORT (ENf NFEFF) 

.J 

1  c 

1  c 

c 

c 

NOW  COMPUTE  SLOPE  BASED  ON  CENTER  POINTS.  WE  WILL  USE 

c 

29%  ,  55%  ,  AND  66%  OF  THE  DATA  ABOUT  THE  MIDPOINT 

c 

OF  THE  SORTED  ARRAY. 

Yj. 

c 

c 

FIRST:  29% 

c 

'A 

139  -  g 

SUM39XY  -  (. 

tc 

SUM30X  •  1. 

SUM39XX  •  0. 

,'s* 

SUM30Y  ■  0. 

SS30XY  •  0. 

SS30XX  -  0. 

DO  230  I  -  F30.L30 

SUM30XY  -  SUM30XY  +  (£N(I)  •  NEWYCN (I ) ) 

SUM30X  •  SUM30X  +  NEWYEN(I) 

t 

%^J 

SUM 3 0XX  -  SUM30XX  ♦  (NEWYEN (I ) »«2) 

O 

SUM30Y  .  SUM30Y  ♦  EN ( I ) 

230 

CONTINUE 

V 

SS30XY  -  SUM  30XY  -  ((SUM 30X*SUM30Y) /D30) 
SS30XX  -  SUN30XX  -  { <SUM30X»*2) /D30) 
SLOPE  30  .  S  LOPE  30  ♦  (SS30XY  /  SS30XX) 
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C  MOW  COUNT  MOW  MANY  REAL  EFFECTS  OCCUR  IN  THE  CALULATIONS 

C  FOR  THE  SLOPE  OF  301  OF  THE  DATA. 

C 

do  j  -  i,r 

DO  K  -  F30,L30 

IF  ( EN  (K )  .  EQ.  EN1(J))  130  -  130  ♦  1 
END  DO 
END  DO 
C 

C  NOW  551 
C 

XS0  -  0 

SUM50XY  -  0. 

SUM S§X  -  0. 

SUM50XX  -  0. 

SUMS0Y  -  0. 

sssaxy  -  0. 

SSS0XX  -  0. 

DO  I  -  FSi,L50 

S  - ‘ 8XY  -  SUM50XY  ♦  (EN  (I )  •  NEWYEN (I ) ) 

-  SUM50X  ♦  NEWYEN  (I) 

-  SUM50XX  ♦  (NEWYEN  (I)  ••2) 

SUMS  -  SUM50Y  +  EN (I) 

END  D 

SS50  SUM50XY  -  ((SUMS0X  *  SUM50Y)  /  D50) 

SS 50a  SUM50XX  -  ( (SUM50X**2)  /  D50) 

StOPEba  -  SLOPE 50  ♦  (SS50XY  /  SS50XX) 

Du.  -  I,P 

J  K  -  F50  , L50 

IF  (EN  (K )  .EQ.  EN1(J))  150  -  1 50  ♦  1 
END  DO 
END  DO 
C 

C  NOW  68% 

C 

170-0 
SUM70XY  -  0. 


SUM70X  -  0. 

SUM70XX  -  0. 

SUM70Y  -  0. 

SS70XY  -  0. 

SS70XX  -  0. 

DO  1  -  P?0,L70 

SUH70XY  -  SUM70XY  ♦  (EN(I)  *  NEWYEN (I ) ) 

SUN70X  -  SUM 7 0X  ♦  NEWYEN (I) 

SUM70XX  -  SUM70XX  ♦  (NEWYEN (I ) **2 ) 

SUM70Y  -  SUM70Y  ♦  EN(Z) 

END  DO 

SS70XY  -  SUM70XY  -  ( (SUM70X  *  SUM70Y)  /  D70) 

SS70XX  -  SUM70XX  -  ((SUM70X**2)  /  D70) 

SLOPE 70  •  SLOPE 70  +  (SS70XY  /  SS70XX ) 

DO  J  •  1,F 

DO  X  -  F70 , L70 

IF  (EN  (K )  .EQ.  ENi(J))  170  -  170  ♦  1 
END  DO 
END  DO 

DO  K  -  1,F 

DO  I  -  1, NFEFP 

IF (EN  (I )  .EQ.  EN1 (X) )  THEN 
COUNT (I )  -  COUNT (I J  ♦  I 
TEST (I )  -  .TRUE. 

END  IF 
END  DO 
END  DO 
DO  1-1 , NFEFF 

IF  (YEN ( I )  .EQ.  0.0000)  THEN 
YEN  (I )  -  0.00001 
END  IF 

COL1 (I)  -  EN(I)  /  YEN  (X ) 

COL2 ( I J  -  COL1  (I)  /  NEWD 
ZHDZE(I)-  COL  2  ( I ) 

NEWE (I )  -  EN (I)  /  NEWD 
ZMDZN(I)  -  NEWE(I)  /  NEWYEN(I) 

END  00 

THE  IF  SLOCK  THAT  FOLLOWS  IS  ONLY  EXECUTED  EVERY 
ONE  HUNDRED  TIMES.  A  COMMENT  OUT  TO  THE  SIDE  WILL 
TELL  WHEHE  THE  BLOCK  STARTS  AND  STOPS. 

IP  (KK  . EQ.  ITEMP)  THEN  I  START  OF  LOOP 

C 

C  FIND  MEDIAN  SIGMA  FOR  FIRST  11  BIASES 
C 

CALL  SORT (NEWE, 11 ) 

DO  1-1,11 

ZMDZEl(X)  -  NEWE  ( I  )  /  YEN  {I  ) 

ENO  DO 

CALL  SORT  (  ZM  DZ  £  1 ,  11) 

ME  D IANS  I GMA  -  ZMDZE1(6) 
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PINO  SLOPE  BASED  ON  30*  ASOUT  THE  CENTER 

HNOZ  30XY-0. 

HNOZ30X»0. 

HNOZ 30XX*0 • 

HNOZ30Y-0. 

HNCZ  3SXY 
HNCZ30X-0. 

HNCZ3IXX-0. 

HNCX3SY-S. 

HEOZ30XY-0. 

HEOZ30X-0. 

HEOZ30XX-0. 

HEOZJSY-0. 

HEC2  3SXY  *0. 

HECZ30X-0. 

HECZ30XX-0. 

HECZ30Y-0. 

DO  J-r3i, L30  | *  *HID  OP  N 

MNOZ30XY  -HNOZ30XY  4  (NEWE(J)  *  MEWYEN  (J )  ) 
HNOZ30X  -MNOZ30X  ♦  NEWYEN(J) 

NNOZ30XX  -HNOZ30XX  4  (NEW YEN  (J )  •  *2 ) 

HNOZ  30  Y  *HNOZ30Y  ♦NEWE(J) 

MNCZ3  0XY  -HNCZ30XY  ♦  (NEWE  (J)  •  YENA(J)) 

HNCZ30X  -HNCZ30X  ♦  YENA(J) 

MNCZ30XX  -HNCZ30XX  4  ( YENA  (J)  *  *2  ) 

HNCZ30Y  -HNCZ30Y  +  NEWE  (J  ) 

END  OO 

DO  J-P30E.J30E  1«*HID  OP  E 

MECZ3  0XY  -HECZ30XY  4  (NEWE(J)  •  YENA(J)) 

HECZ30X  -MECZ30X  4  YENA(J) 

HECZ30XX  "HECZ  3  0XX  4  (YENA(J)*42) 

HECZ30Y  -HECZ30Y  4NEWE(J) 

HEOZ3  0XY  -MEOZ30XY  4  (NEWE  (J)  •  NEWYEN  ( J )  ) 
HEOZ30X  *HEOZ30X  4  NEWYEN  (J) 

MEOZ30XX  -HEOZ30XX  4  (NEWYEN  (J )•  »2 ) 

HEOZ30Y  -MEOZ30Y  4  NEWE(J) 

ENO  DO 

SSMNOZ30XY-0. 

SSHNOZ30XX-0. 

SSHNCZ30XY-0. 

SSHNCZ30XX-0. 

SSHEOZ30XY-0. 

SSMEOZ30XX-0. 

SSMECZ30XY-0. 

SSMECZ30XX-0. 

SSHNOZ  30XY  -  MNOZ30XY  -  ({HNOZ30X  •  HNOZ 30Y) /D30) 
SSHNCZ30XY  -  HNCZ30XY  -  (  (HNCZ3  0X  •  HNCZ  30Y )  /D30  ) 
SSHNOZ  30XX  -  HNOZ30XX  -  (  ( HNOZ  3  0X  ••  2  ) /D  30 ) 
SSHNCZ30XX  -  HNCZ30XX  -  ( (HNCZ3 0X • *2 ) /D 30 ) 

SSHECZ  30XY  -  HECZ30XY  -  ((HECZ30X  •  HECZ  30YJ/D30) 
SSHECZ30XX  -  HECZ30XX  -  (  (HECZ 3 0X •  *2 )  /D30  ) 


i  "  HLOZ30XY  -  ( ( HEOZ  31 
SSHEOZ30XX  -  HEOZ30XX  -  ((HE0Z3I 
HNOZ30-0. 

HNCZ30h0. 

HEOZ30.0. 

NECZ30-0. 

HNOZ  30  •  (SSHNOZ  30XY  /  SSHNOZ  30XX) 
HNCZ30  -  (SSHNCZ30XY  /  SSHNCZ3  0X 
HEOZ  30  -  (SSHEOZ  30XY  /  SSHEOZ  30X 

HECZ30  •  (SSHECZ 3 0XY  /  SSHECZ 3  0XX ) 
HNOZS0XY-0. 

HNOZ50X-0. 

HNOZ50XX-0. 

HNOZ50Y-0. 

HNCZ50XY-0. 

HNCZ50X-0. 

HNCZ50XX-0. 

HNCZ50Y-0. 

HEOZ50XY.0. 

HEOZ50X-0. 

HEOZ  50XX.0. 

HEOZ5ZY-0. 

HECZ  S0XY  «0 . 

HECZ  5M  -  0. 

HECZi0XX-0. 

HECZS0Y-0. 


-  ( ( HEOZ  30X  •  HEOZ30YJ/D30) 

-  ((HEOZ30X**2)/D30) 


o  o  o 


c 

C  FIND  SLOPE  BASED  ON  50%  OF  THE  CENTER 

C 

DO  K-F50,LS0  |**MlD  OF  N 

HNOZ50XY-  HNOZ50XY  ♦  (NEWS  (K)  •  NEWYEN(K)) 
MNOZS0X  -  MNOZS0X  ♦  NEW  YEN  (K  ) 

MNOZS0XX  -  MNOZ50XX  ♦  (NEWYEN (X) **2) 
MNOZS0Y  -  MNQZ50Y  ♦  NEWE(K) 

MNCZ50XY-  HNCZ50XY  ♦  (NEWE  (K)  *  YENA (K) ) 
MNCZS0X  -  MMCZ50X  ♦  YENA(K) 

MNCZS0XX-  MNCZ50XX  ♦  (YENA(K)**2) 

MMCZSBY  -  MNCZ  50Y  ♦  NEWE(X) 

ENO  DO 

DO  X-P50E, JS0E  i *  *M  ID  OF  E 

MECZ  50XY  -  HECZ  5 f  XY  ♦  {NEWE  (K)  •  YENA  (X )  ) 
MECZSfX  -  MECZ  S0X  ♦  YENA(X) 

MECZ50XX  •  MECZ  5  0  XX  ♦  (Y£NA(X)*«2) 

MECZ50Y  -  MECZ50Y  ♦  NEWE(K) 

MEOZS0XY-  MEOZ50XY  ♦  (NEWE  (X)  •  NEW  YEN  (X )  ) 

MEOZ50X  •  HEOZ50X  +  NEWYEN(K) 

HEO250XX-  MEOZ50XX  ♦  (NEWYEN  (K) **2) 

MEOZS0Y  -  MEOZ50Y  ♦  NEWE(K) 

END  DO 

SSKNOZ50XY-0. 

SSMNOZ50XX-0. 

SSMNCZ50XY-0. 

SSMNCZ50XX-0. 
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794 


SSMEOZ50XY-0. 

SSMEOZ50XX-0. 

SSMECZ  S0XY  -0 . 

SSHECZS0XX-0. 

SSMNOZ50XY  -  M  NO  Z  5  0  X  Y  -  {(MNOZ50X  •  MNOZ S0Y ) /D50 ) 

SSMECZ  5  0XY  -  MECZS0XY  •  ((f"  CZ50X  •  MECZ  50Y ) /D50 ) 
SSMNOZ50XX  -  MNOZ50XX  -  ( {MNOZ 50X **2 ) /D50 ) 

SSMNCZ50XX  -  MNCZ50XX  -  ( (MNCZ50X*  *2} /D50 ) 

SSMNCZS0XY-  MNCZS0XY-  ((MNCZ50X  •  MNCZ  50  Y ) /D50  ) 

SSMECZ 50XX-  MECZ50XX-  ( ( M ECZ 5 0X • *2 ) /D 50 ) 

SSMEOZ50XY-  MEOZ50XY-  ( ( mEOZ  50X  •  MEOZ 50Y ) /DS0 > 
SSMEO250XX-  MEOZ50XX-  ( (MEOZ 50X**2) /D50) 

MNOZ  50  *0 . 

MNCZ50-0. 

MEOZ 50-0. 

MECZ  50-0. 

MNOZ  50  -  (SSMNOZ50XY  /  SSMNOZ50XX)  |*SLOPE  MID  N  OZ 

MECZ  50  -  (SSMECZ50XY  /  SSMECZ  50XX)  l*SLOPE  MID  E  CZ 

MNCZ  50  -  (SSMNCZ50XY  /  SSMNCZ50XX)  I*SLOPE  MID  N  CZ 

MEOZ 59  -  (SSMEOZ50XY  /  SSMEOZ50XX)  ISLOPE  MID  t  OZ 

OPEN (UNIT-9,  TYPE -'NEW'  , FILE- ‘SIMON. DAT' ) 

WRITE (9, 793 ) KX, NFEFF, F,POSEFF 
WRITE  (9#  794  )MAG  (1 ) 

WRITE (9,798) 

FORMAT  (IX,  'RUN  •  ,  15,  3X,  '  K-*  ,  15,  3X,  1  REALS-*  ,15,  3X, 

♦  EFFECTS-' ,15) 

FORMAT (IX, 'MAGNITUDE  OF  EFFECT  IS',F7.4//) 

DO  I -1, NFEFF 

IF (TEST (I ) ) th«n 

WRITE  (9,79i)EN(I)  ,  NEWE  (I )  ,  ZM  DZ  E  (I  )  ,  ZMDZN  (I ) 

ELSE 

WRITE  (9, 792  )EN  (I  )  ,  NEWE  (X  )  ,  ZMDZE  (I )  , ZMDZN  (I  ) 

END  IF 

END  DO 

WRITE (9, 795 )MEDIANS  XGMA 
WRITE (9, B50) MNOZ 30 
WRITE (9, 051) MNCZ 30 
WRITE (9, 052) MEOZ 30 
WRITE (9, «5 3) MECZ 30 
WRITE (9, 054) MNOZ  50 
WRITE (9,855) MNCZ 50 
WBITE(9,056)MEOZ50 
WRITE (9,857) MECZ  50 


WRITE  SLOPE  SCORES  OUT  FOR  FREQ.  DIST. 


WRITE  (II ,♦) MEDIANS  I CM A, MECZ 30, MECZ  50 
C 

C  WRITEOUT  DATA  FOR  BMDP 

C 

DO  I-I, NFEFF 

MAT ( l  , NEW K )  -  NEW  E  {  I  ) 

ENO  no 
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7*5 

FORMAT  (« 

MED. 

OF 

EMD/EE*  ,P15. 5) 

•  50 

FORMAT (• 

MID. 

OF 

N 

0130', PIS. 5) 

051 

FORMAT (• 

MID. 

OF 

N 

CZ30',ri5. 5) 

052 

FORMAT ( • 

MID. 

OF 

E 

OZ30',P15.5) 

853 

FORMAT {• 

MID. 

OF 

E 

czia'.ris.S) 

854 

FORMAT  (• 

MID. 

OF 

N 

OZS0 1  ,P1  5.  5) 

855 

FORMAT  (• 

MID. 

OF 

N 

CZ  50 ' ,P1S. 5) 

856 

FORMAT ( • 

MID. 

OF 

E 

OZ50*,P15.5) 

857 

FORMAT (• 

MID. 

OF 

e 

CZ  50  ' ,F15.  5) 

798 

FORMAT (• 

MO 

♦  2MD/ZN ') 

C  CLOSE  (UNIT -9) 

ITEMP  -  XTEMP+lif 


NEWK  -  NEWK *1 
END  ZP 
C 

791  FORMAT (F15. 5 , *R'#3P15. 5) 

C 

C  SUM  UP  FOR  MEANS  AND  VARIANCES 
C 


ZMD  ZMD/ZE 


I  END  OF  LOOP 


DO  45  X *1 , NFEFF 
E(X)-E  (I)4EN  (I) 

El  (I )  -  E 1  (I )  ♦  EN  1(1) 

ESS  (ll«e$S<l)4CM{l)**2 
45  CONTINUE 

T30  -  T30  ♦  130 

T50  -  T$0  ♦  IS® 

T7 0  -  T7®  ♦  170 

C 

C  INC.  COUNTER  FOR  THE  50  COLUMNS  XH  MATRIX  MAT 

C 

100  CONTINUE 

C 

C  NOW  WRITE  MAT  OUT  FOR  BMDP  ANALYSIS 
C 

OPEN (UN IT-15, TYPE- 'NEW*  , NAM £> »S IMONBMD. OAT • , RECORDS  IX C-8®0) 
DO  J ml'NPEFr 

WRITE  (15,  792)  (MAT  (J ,  I )  ,  I -1 ,  50  ) 

END  DO 

CLOSE  (UNIT-15) 

C 

C  GET  MEANS  AND  VARIANCES 

doP120  j-l.ttrtrr 

£  (J )  -E  (J)/REP 
El  (J)  -  El  (J)  /  REP 
NEWE  (J)  -  E  (J)  /  NEWD 
ZHDZN(J)  -  NEWE(J)  /  NEWYEN(J) 

IE  (YEN  (J)  .EQ.  •.)  YEN  (J )  -  .aim 
COLl(J)  -  E  (J  )  /  yen  (J ) 

COL2(J)  •  COH  (J)  /  NEWD 


ESS(J)-ESS(JJ/REP-E  <J)**Z 
ESS(J)«  SQRT(ESS(J))/N£WO 
12f  CONTINUE 

SLOPE3B  -  (SLOPE  30  /  REP)/  NEWD 
SLOPE50  -  (SIOPE5B  /REP)  /  NEWD 
SLOPE70  «  (SLOPE 7a  /  REP)/  NEWD 

c 

OPEN  (UN  IT-«  . TYPE* 'NEW'  .  P I LE -  ■  WES  .  DAT  •  ) 

OPEN  (UNIT-7.TYPE-'NEW  ,PI  LE- '  ZMD.  DAT’  ) 

c 

WRITE (6,753)  NSCELL 

751  PORMAT(*  NUMBER  OP  SUBJECTS  PER  CELL  IS*,33) 

WRITE (6, 755 )  NR 

755  TORN  AT  ('  NUMBER  OP  RUNS  IS’, IS) 

WRITE  (t,7Sa)  P 

78P  PORMAT  ('  NO.  REAL  EPFECTS  IN  THIS  DES1CN  IS’, 15) 

WRITE  (6, 7 85)  (MAO  (J)  ,J-1.P) 

WRITE (6,789) 

WR ITE ( 6 , 7  90  ) 

DO  130  J«l,NP£Pf 

130  WRITE  (6 , 550  )  J ,  E  (J  )  .COUNT  (J  ),  NEWE  (J  ),  YEN  (J  )  , 

X  ETP  (J)  ,  COLl  (J)  ,COL2  (J)  ,  ZMDZN  (J)  ,  ESS  (J)  ,  YEN  2  !J) 

C 

785  FORNAT(10X,  'MAGNITUDE  OF  THE  CFFECTS  ARE  '  ,8F7.«) 

790  FORMAT  (1  0X,  ' _ _ _____ 

X _  '  ) 

78  9  FORMAT  (1  0X,  '  M.O.  EFFECTS  ZMD  ZE 

X  MD/ZE  ZMD/ZE  ZmD/ZN  STD.  sd  .  E.N.  «d.'l 

550  FORMAT  (10X,I3,F8.3,lX,n,2x,F8.3,2X,F8.3,lX, 

X  *  C  ,  12,  '  )  1  ,  3F10.  3,  2F10.  3| 

551  FORMAT  (I  0X,  I  3,  F8.  3,  IX,  F8.  3,  IX,  •  (•  ,  12,  '  )  1  . 2F10. 3) 

792  FORMAT (50F1 5. 5) 

DO  K-l, NFEFF 

WRITE  (7,  •)  NEWE  (X) 

EN 0  DO 

CLOSE  (UN  IT-7) 

C 

STOP 

ENO 
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C 

SUBROUTINE  SORT (X,  M) 
DIMENSION  X  (127) 

M-N 

1  K-« 

DO  2  1-2, H 
J-I-l 

IP(X(J)  •  LE«  X(I))  CO  TO  2 
K«sJ 

S-X(I) 

X(I)-X  (J) 

X(J)-S 

2  continue 

ir(K  .eo.  ijRETURN 

M  »K 

CO  TO  1 
END 

c 


c 

c 

c 

SUBROUTINE  R NORM 01  (H,X,SE£D) 

C 

C  TO  GENERATE  RANDOM  NORMAL  NUMBERS  PROM  UN.  RANDOM 

C  NUMBERS  BY  BOX  AND  MULLER (1958)  METHOD.  If  Rl  AND  R2  ARE 

C  2  UN.  RAND  NUMBERS,  THEN 

C  Z1 “SORT (-2. *LOG (RX) )*COS (2*PI#R2) 

C  Z  2 “SQRT  (-2 •  #LOG  (R 1 )  )  *S  IN  (2*PI*R2) 

C  ARE  A  PAIR  FORM  N(0,l)  . 

C 

IMPLICIT  REALM  (M) 

DIMENSION  X  (1 28 ) 

DO  50  I  -1 ,  N,  2 

Ri -MTH  $ RAN DOM (SEED ) 

R2-MTH$RANDOM (SEED) 

X (I)-SQRT(-2.*LOG(Rl)} *COS (6. 2831  853  *R  2) 
X(H-l)-SQRT(-2.*LOG<Rl))  *SIN  (6. 2  8 31  8  53*R2 ) 

50  CONTINUE 
RETURN 
END 
C 
C 
C 
C 


C 

C 

SUBROUTINE  SC OR (J,N, SCORE, VAR) 

C 

C  COMPUTES  THE  E(x)  OP  THE  JTH  OP  N  RANKED  NORMAL 
C  SCORES. 

C  SCORE  RETURNS  WITH  THE  EXPECTED  NORMAL  SCORE. 

C  VAR  RETURNS  WITH  THE  VARIANCE  OP  THE  NORMAL  SCORE. 

C  COMPUTE  THE  CONTENT  OP  THE  INTEGRAL  LOG  FORM 
C  -  PACL  (N) -PACL  (J-l )  -PACL  (N-J) 

C 

C  GET  THE  APPROX.  NORMAL  SCORe 
C 

AM  -  SCR  (J  ,  N) 

ANN  -  AN 
J  ANN  -  ANN-.  1 

I f  (FUN (ANN,C, J, N)  .GT.  0.)  GO  TO  3 
R0OT  ■  ANN 

A  A  MM  m  A  UM  4  1 

If  (PUN (ANN,C, J,N)  .GT.  0.)  GO  TO  4 
RTOP  -  ANN 
RNG  -  AN-R0OT 
XU  -  AN 

ru  -  XU *PUN (XU,C, J,N) 

W  -  RNG/160. 

PRT1  -  0. 

PRT1V  -  0. 
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C  START  THE  FIRST  SIMPSON  INTEGRATION 

C 

C 

If  XL  -  XU-W 

XN  -  (XU ♦XL)  /2. 

FN  •  XN* FUN (XN ,  C, J,  N) 

FL  •  XL*FUN(XL,C,J,N) 

FI  -  PRT1+(FL+4.*FN*FU) *W/6. 

FRTIV  -  PRT1V+  (XL*FL*4 .  *XM*FM+XU*FU)  *W/f. 
IF  (PRTl  .  EQ.  PI)  GO  TO  15 
W  •  11*1.1 
XU  -  XL 
FU  -  FL 
PRTl  -  PI 
CO  TO  If 
15  CONTINUE 

RNC  -  RTOP  -  AN 
XL  •  AN 

FL  -  XL* PUN (XL,C,J,N) 

W  «  RNG/lff. 

PRT2  -  0. 

C 

C 


C 

PRT2V  -  0. 

C 

C  START  SECOND  SIMPSON  INTEGRATION 
C 

20  XU  -  XL+W 

XM  -  (XU+XL)/2. 

FM  -  XM*FUN (XM,C, J,N) 

PU  -  XU* PUN (XU, C,  J,  N) 

P2  -  PRT2+(PU+4.*FM+PL)*W/6. 

PRT2V  -  PRT2V+ (XU*PU+4 . *XM*FM+XL*FL) *W/6. 

IP  (PRT2  • EQ.  P2)  GO  TO  25 
W  -  W*l.l 
XL  -  XU 
FL  -  PU 
PRT2  -  P2 
GO  TO  20 
25  CONTINUE 

SCORE  -  PRT1+PRT2 
SCR 2  *  PRTIV+PRT2V 
VAR  -  SC R 2-5C OR  E  *SC OR  E 
RETURN 
END 
C 
C 

c 

c 

FUNCTION  rUN(X,C,J,W) 

C 

C  COMPUTES  VALUE  OP  FUNCTION  TO  BE  INTEGRATED,  FIRST  IN  LOG 
C  PORN,  THEN  IN  DECIMAL  FORM  IF  LARGE  ENOUGH. 

C  C  -  N  J/(J-1 )  |/{N-J )  |  -  THE  CONSTANT  OF  INTEGRATION 
C  FUN  •  C*P**(N-J)  *Q**(J-I)  *Z 

C  WHERE  P  IS  THE  PROPORTION  BELOW  Q  IS  THe  PROPORTION  ABOVE 
C  X  ON  THE  NORMAL  CURVE;  AND  2  IS  THE  ORDINATE  OF  THE  NORMAL 
C  CURVE  AT  X. 

C 

DOUBLE  PRECISION  X,  POPZ  ,  P,  Q,  XJ,  XN,  FUN,  FL 
XJ  •  J 
XN  -  N 
P  -  POFZ(X) 

0  -  l.Dfl-P 
FUN  -  fl.Df 

IF  (P  .LE.  f.D0  .OR.  0  .  L£  •  0.  D0)RETURN 

FL  -  C+DLOG  (P)  *  (XN-XJ )  +DLOG  (Q)  *(XJ-1.D0)-X  *X/2.  D0-0. 918  93  8S33D0 
IF  (FL  .LT.  -40. ) RETURN 
FUN  -  DEXP(FL) 

RETURN 

END 

C 

C 
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c 

FUNCTION  FACL(K) 

C 

C  COMPUTES  THE  NATURAL  LOG  OF  Kl 
C  ABOVE  36  USES  STIRLINGS  APPROXIMATION 
C 

IP  (K  .GT.  36)  GO  TO  20 
FACL  -  0. 

IF  (K  .LE.  1)RETURN 
DO  10  1-2, K 
X  -  I 

10  FACL  -  FACL+ALOG  (X) 

RETURN 
20  X  -  X 

FACL  -  .918938533*ALOG(X) • (X*. 5)-X*l ./( 1 2. *X) 

RETURN 

END 

C 

C 

FUNCTION  SCR(J,N) 

C 

C  RAPID  APPROXIMATION  TO  THE  JTH  OF  N  EXPECTED  NORMAL  SCORES. 
C  USES  BLOMS  ALGORITHM  WITH  CORRECTIONS  PROPOSED  BY 
C  HARTLEY,  BIOMETRIKA,  1961,  48,  151-165. 

C 

XJ  -  J 

XN  -  N 

X  -  ALOG10  (XN) 

IF  (N  .GT.  400)  GO  TO  10 
A 1  -  .315065*. 057974*X-.009776*X*X 
A2  -  . 327511*. 058212*X-.007909*X*X 
CO  TO  20 

10  A 1  -  .3752*. 00976 *X*. 00008 *X*X 
A2  -  .3866*. 01418*X*. 00029 *X*X 
20  A  -  A2 

IF  (J  .EQ.1.D0  .OR.  J  .£Q.  N)  A-Al 
SCR  -  Z  INV  (  (XJ-A)/(XN-2.*A*1.)  ) 

SCR  -  -SCR 
RETURN 
END 
C 


c 

FUNCTION  ZINV(P) 

C 

C  QUICK  APPROXIMATION  TO  THE  INVERSE  OF  THE  NORMAL  DIST.  FUNCTION 
C  EMERSON,  BRMI,  1979,11,  397-398. 

C 

SGN  -  -1. 

Q  -  P 

IP  <0-.5)2,2,l 

1  SGN  -  1. 

Q  -  l.-Q 

2  Z  *  SQRT  (—2.  *ALOG  (Q)  ) 

W  -  Z*(. 18926*2*. 001308) 

W  ■  l.+Z*(1.432788*W) 

0  -  .802853*2*. 010328 
W  -  (2.  51 551 7*2  *Q)  /W 
ZINV  -  (Z-W)*SGN 
RETURN 
END 
C 
C 

DOUBLE  PRECISION  FUNCTION  POFZ(XX) 

C 

C  NORMAL  PROBABILITY  INTEGRAL,  -INFINITY  TO  X|  X>0 
C  ALGROR ITHM  26.2.11  P.932  ZELEN  SEVERO 
C 

DOUBLE  PRECISION  X,  XX,  S,T, C,  XN ,  SN 
X-  DABS (XX) 

POFZ  -  1.D0 

IF  (X  .GT.  8.35)  GO  TO  20 
S  -  X 
T  -  X 
C  -  X*X 
XN  -  0.D0 
5  XN  -  XN  *  1 .  D0 

T  -  T*C/(2. D0*XN*1 . D0) 

SN  -  S+T 

IF  (SN  .EQ.  S)  GO  TO  10 
S  -  SN 
CO  TO  5 

10  POPZ  -  . 5D0*. 398942280401 4327D0*DEXP (-C/2. D0) *S 
20  IP  (XX  . LT.  0.D0)  POFZ-1.  D0-POFZ 
RETURN 
END 
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COMPUTER  PROGRAM  FOR  GENERATING  EXPECTED  VALUES  OF 
NORMAL  AND  HALF-NORMAL  ORDER  STATISTICS 


David  G.  Weinman,  Ph.D. 


This  program  was  written  in  VAX-BASIC  and  run  on  the  Hollins  VAX  11-780 
computer.  The  following  program  is  for  generating  half-normal  order 
statistics.  To  generate  normal  order  statistics,  delete  the  two  references  to 
absolute  values  (ABS)  in  line  230  and  change  comments  regarding  "half-normal" 
to  "normal." 


100  Rem  —  Generate  hair— normal  order  statistics 

110  DECLARE  INTEGER  GAP,  I,  J,  N,  NREP,  NXT ,  P,  REP.  SDRTFLAG.  T.  X 
DECLARE  REAL  A<128),  ASUM<128),  ADD,  Nl,  TEMP,  MEAN,  STD 
RANDOMIZE 
NREP  =  5000 
120  FOR  N  =  20  TO  31 
130  Nl  =  N/2 

140  IF  I NT ( N 1 )  =  Nl  THEN 

K1  =  Nl 
ELSE 

K1  =  Nl  +  1 
END  IF 
Z*  =  STR*(N) 

DATA*  =  "FACULTY: CDAVE. SIM0N3HALFN"  +  Z*  +  ".DAT" 

150  FOR  I  =  1  TO  N 

ASUM(I)  =  0 
NEXT  I 

160  FOR  REP  =  1  TO  NREP 

GOSUB  200  !  Generate  N  ha  1 f-norma 1 (0,  1)  values 

GOSUB  400  !  Sort  the  values 

FOR  I  =  1  TO  N 

ASUM ( I )  =  ASUM ( I )  +  A ( I ) 

NEXT  I 

170  NEXT  REP 

GOSUB  600  !  Calculate  mean  of  order  statistics  and  print 

180  NEXT  N 

GO  TO  Finish 

190  |  |  I;;  |  >  i  i  i  i  <  i  >  i  ><  '  i  i  i  <><>  i  <>  i  i  i  i  i  i  i  i  i  i  i  i  i  i  i  i  i  i  i  i  i  i  i  i  i  i  i  i  i  i  i 
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200 

210 

220 


230 


240 

250 

300 

400 

410 

420 

430 

440 


!  Generate  N  half-normal  variates  (Polar  Marsaglia  method) 
FOR  I  =  1  TO  K1 

VI  =  2  *  RND  -  1 

V2  =  2  *  RND  -  1 

R2  =  V1*V1  +  V2*V2 

IF  R2  >  1  THEN  220 

Y , =  SQR (  (-2  *  LOG ( R2 )  )  /  R2) 

A ( 2* I - 1 )  =  ABS ( V 1  *  Y) 

A ( 2* I )  =  ABS ( V2  *  Y) 

NEXT  I 
RETURN 

I  •  I  I  I  I  I  I  I  I  !  I  1  I  I  I  I  I  I  I  I  I  I  1  I  I  1  1  I  1  »  I  l  I  I  I  I  I  I  I  I  I  I  I  I  I  I  I  I  I  I  !  I  I  I  I  I 

!  Subroutine:  Sort  the  array  A 
GAP  =  N 
WHILE  GAP  >  1 

GAP  =  GAP  /  2 

!  Put  numbers  GAP  positions  apart  in  order. 

TOP  =  N  -  GAP 
SORTFLAG  =  O 
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k\* 

WHILE  SORTFLAG  = 

0 

1  / 

SORTFLAG  =  1 

■  V 

FOR  I  =  1  TO  TOP 

yfiki 

NXT  =  I  + 

GAP 

m 

P 

IF  A(I)  > 

A (NXT)  THEN 

** 

-•  / 
f  .  -  . 

SORTFLAG  =  0 

i 

TEMP 

=  A  (  I  ) 

i 

A  (  I  ) 

=  A ( NXT ) 

. 

A (NXT ) 

=  TEMP 

S  ,* 

END  IF 

t  *  M 

NEXT  I 

NEXT 

NEXT 

0'.'-  - 

\*V 

9  m. 

% |jr* 

450 

RETURN 

W  : 

<  ■  . 

500 

1  1  1  1  1  1  1  1  1  1  1  1  1  1  1  1  I  1  I 

1  1  1  1  1  t  1  1  f  1  1  1  1  1  1  1  1  1  1  1  1  1  1  1  1  1  1  1  1  1  1  1  1  1 

i  i  i  i  i  i  i  i 

600 

!  Calculate  mean  of 

order  statistics 

610 

OPEN  DATA*  FOR  OUTPUT  AS  FILE  #1 

;•> 

620 

FOR  I  =  1  TO  N 

i 

MEAN  =  ASUM ( I ) 

/  NREP 

;  > 

PRINT  #1,  MEAN 

rP 

p.-.v 

630 

NEXT  I 

r? 

640 

CLOSE  #1 

t. 

600 

R E  TUR  N 

v 

<b90 

i  i  i  i  i  i  i  i  t  i  i  i  i  i  i  i  \  »  i 

1  »  »  »  1  1  \  1  1  t  t  »  1  »  »  »  1  »  1  1  1  1  1  »  1  1  1  1  »  »  »  1  1  1 

1  l  l  l  i  1  l  l  l  V*, 

»*,v. 

700 

Finish:  END 

\ 

APPENDIX  E 


DEVIATION  OF  STANDARD  ERROR  OF  THE  CONTRASTS 


The  standard  error  of  the  contrasts  (s  >  is  the  same  as  the  standard 

c 

error  of  the  mean  differences.  '  The  general  equation  for  this  is: 


[ox  =  o2] 


n/2  n/2 


sc  =  \  20  a 


Sc  %  02 


sc  oa 
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PREDICTING  R-SPILLOVER  USING  CONTRASTS  AND  CONTRAST  STANDARD  DEVIATIONS 

David  G.  Weinman,  Ph.D. 

[The  author  asked  Dr.  Weinman  to  Investigate  whether  or  not  the  contrast 
standard  deviation  at  each  rank  (see  Table  1  Eor  example)  could  be  used  to 
estimate  the  R-splllover.  The  R-spillover,  it  was  felt,  could  be  useEul  to  an 
investigator  who  wants  to  decide  whether  It  Is  likely  that  a  smaller  contrast 
may  occur  at  a  rank  below  where  other  techniques  had  determined  the  contrast 
was  likely  to  be  a  real  one.  In  the  discussion  below.  Dr.  Weinman  explains 
what  might  be  done,  how  complex  it  can  be  to  do  It  correctly,  and  the 
theoretical  limits  oE  this  approach.] 

For  any  sample  size  n  >  2  Erom  a  normal  population,  the  longest  and 
smallest  order  statistics  do  not  have  a  normal  distribution.  However,  we 
attempt  here  to  estimate  spillover  by  using  an  assumption  oE  normality, 
because  a  normal  distribution  oEten  gives  reasonable  (approximately  correct) 
results. 

Consider  the  situation  oE  n  ■  31  eEEects,  and  suppose  we  add  1.67o  to 

eight  oE  these  eEEects.  Our  1.67a  is  equal  to  [1.67  *•  4/32]s  ,  where 

c 

sc  is  the  standard  error  of  a  contrast,  so  we  actually  add  [1.67  +  y 4/32], 
or  4.723  to  each  of  eight  contrasts. 

Then  we  have  31  effects,  of  which  23  are  "error"  effects,  and  8  are  "real" 
effects.  The  23  error  effects  are  distributed  as  23  order  statistics  (from  a 
sample  of  23)  from  a  normal  (0,  1)  distribution,  while  the  8  real  effects  are 
distributed  as  the  order  statistics  from  a  sample  oE  8  from  a  normal 
[(1.67  -r  /  4/32)  ,  1.00]]  distribution. 

From  our  simulation,  the  means  and  standard  deviations  are  as  follows  for 
the  longest  3  of  23  order  statistics  and  the  smallest  3  of  8  order  statistics, 
with  4.723  added  to  the  three  smallest.  We  use  x  for  the  error  effects  and  y 
for  the  real  effects. 
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With  some  5000  replications,  we  would  expect  to  find  7.0,  21.5,  and  174.0- 
real  effects  In  positions  21,  22,  and  23  respectively.  On  one  run  we  found 
11,  43,  and  186  real  effects  In  those  positions.  To  summarize. 

Expected  Number  Actual  Number 


Position 

of  Real  Effects 

of  Real 

21 

7.0 

11 

22 

21.5 

43 

23 

174.0 

186 

The  results  for  positions  21  and  23  are  encouraging,  but  the  prediction  at 
position  22  is  off  by  100%. 

This  case  —  8  real  effects  of  size  1.67a  —  is  one  of  the  cleanest 
situations  to  deal  with,  because  1.67a  is  a  large  effect.  With  smaller  real 
effects,  many  more  probabilities  must  be  calculated  and  the  expected  number  of 
real  effects  is  a  poorer  estimate  of  the  actual  number  of  real  effects  found 
in  simulations. 
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VA 

i 

Order 

Contrast 

Contrast  Std. 
Deviation 

Order 

Contrast 

Contrast  Std 
Deviation 

4  • 

“  _  ■> 

X21 

1.21934 

0.33574 

yi 

3.29987 

0.61065 

.-S» 

X22 

1.48096 

0.27804 

y2 

3.87125 

0.48929 

*\V 

;  X23 

1.91716 

0.48899 

y3 

4.25065 

0.44870 

The  spillover  rate  from  real  effects  Into  error  positions  then  Involves 
the  probabilities  P(y^  <  x^)  that  some  is  less  than  some  x^. 

Dropping  the  subscripts  i  and  j,  we  have 


P(y  <  x)  =  P(y-x  <  0) 


U  -  U 
x  y 

=  P  (z  <  a2  +  a2 
x  y 


where  z  is  a  standard  N[0,1]. 

For  i  =  1,2  and  j  =  22,23,  these  probabilities  are: 


P(y  <  x  )  =  .0384 

1  23 

P(Yx  <  X22}  =  '°057 
P(Yi  <  X21}  =  *°°14 

P(Y3  <  x23)  =  .0024 

PCy  <  x_  )  =  .0000 

2  /./. 


Then,  assuming  independence,  we  have  the  following  probabilities  of  real 
effects  spilling  over  into  positions  21,  22,  and  23. 


Position 


P  (Real  effect  in  position) 


.0014 

.0043  (  =  .0057  -  .0014) 

.0348  (=  .0384  -  .0057  +  .0024) 
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PROGRAM  FOR  GENERATING  GUARDRAILS 
David  G.  Weinman,  Ph.D. 

This  program  was  written  in  VAX-BASIC  and  run  on  the  Hollins  VAX  11-780 
computer.  The  following  program  is  for  generating  guardrails  for  half-normal 
plots.  To  generate  guardrails  for  normal  order  statistics,  delete  the  two 
references  to  absolute  values  (ABS)  in  line  250  and  change  comments  regarding 
"half-normal"  to  "normal." 


100  !  PROGRAM  GUARD  —  Obtain  guardrails  for  half-normal  plots 
!  Generate  half-normal  variables 

!  Find  percentiles  of  largest  order  statistic  /  slope  estimate 
110  DECLARE  INTEGER  GAP,  I,  J,  Kl.  M.  N,  NREP,  NXT.  P,  REP,  S0RTFLAG 
DECLARE  REAL  N0RM<64>,  A<&4).  LARGE < 10000 ) .  R2,  VI.  V2,  Y 
DECLARE  REAL  A05,  A10.  A20.  A40.  B.  Nl.  R,  SSX,  SXY,  TEMP 
RANDOMIZE 


120  NREP  =>  5000 

!  Open  output  file  and  print  header 
OUTFILES  =  "FACULTY:  CDAVE.  S IMON 1 ALPHA3 1 .  OUT" 
OPEN  0UTF1LE*  FOR  OUTPUT  AS  FILE  til 


130  PRINT  til, 
PRINT  *1, 
PRINT  »1 
PRINT  ttl, 
PRINT  ttl, 
PRINT  ttl, 
A*  » 


"Half  Normal" 

"Number  of  replications  “i  NREP 

"  Values  of  alpha" 

"  tt  .05  .10  .20  .  40" 


"  tttt  «».  tttttt  tttt.  tttttt  tttt.  tttttt  ttt*.  tltttt" 


!  Main  program:  N  is  the  number  of  effects  being  considered 
!  Polar  Marsaglia  method  generates  an  even  number  of  values, 
!  so  determine  Kl  and  generate  2K1  values, 
t  Open  input  file  and  read  ha'f-normal  values. 

140  FOR  N  -  31  TO  20  STEP  -1 


Nl  -  N  /  2 
IF  INT(Nl)  -  Nl  THEN 
Kl  -  Nl 
ELSE 

Kl  -  Nl  ♦  1 
END  IF 


P  •.* 


INFILE»  =  "FACULTY:  CDAVE.  S IMON 1HALFN" 
OPEN  INFILE*  FOR  INPUT  AS  FILE  42 
FOR  I  *  1  TO  N 

INPUT  42,  NORM  < I > 

NEXT  I 
CLOSE  42 


♦  STRS(N)  ♦  ».  DAT" 


R 
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130 


FOR  REP  -  1  TO  NREP 

COSUB  200  !  Generate  N  half-normal (O.  1  >  value* 

COSUB  400  !  Sort  array  A  and  save  largest  value 
GOSUB  300  !  Calculate  elope  and  divide  largest  value  by  it 

160  NEXT  REP 

GOSUB  600  !  Sort  the  values  of  LARGE 
GOSUB  800  !  Print  results 
170  NEXT  N 
CLOSE  #1 

180  GO  TO  Finish 

190  1  f  |  t  i  |  i  t  |  i  i  i  i  |  i  ;  i  i  i  t  i  i  i  i  i  i  i  i  t  i  i  t  *  i  i  i  i  i  i  i  i  i  i  i  i  i  i  i  i  i  i  i  i  i  i  i  i  i 

200  !  Generate  N  half-normal  variates  (Polar  Marsaglia  method) 

!  as  given  in  Morgan:  Elements  of  Simulation 
230  FOR  I  =  1  TO  HI 
240  VI  »  2  *  RND  -  1 

V2  =  2  *  RND  -  1 

R2  -  V1*V1  +  V2*V2 

IF  R2  >  1  THEN  240 

250  Y  *>  SQR  (  (-2  *  L0G(R2>  >  /  R2) 


A< 2* I  —  1 )  =  ABS (VI  *  V) 

A ( 2* I )  =  ABS ( V2  *  Y) 

260  NEXT  I 
270  RETURN 

390  i  )  ;  i  <  *  >  i  i  t  i  i  i  i  •  i  i  i  i  i  t  *  i  i  i  i  i  •  i  <  i  i  >  i  i  <  i  i  <  •  i  i  t  i  i  i  i  t  i  i  i  *  i  i  i  t  i  i  i  i 

400  !  Subroutine:  Sort  ti  array  A  (Shellsort) 

410  GAP  »  N 

420  WHILE  CAP  >  1 

430  GAP  =  GAP  /  2 

440  !  Put  numbers  GAP  positions  apart  in  order. 

TOP  =  N  -  GAP 
SQRTFLAG  =  0 
WHILE  SORTFLAG  =  O 
SORTFLAG  =  1 
FOR  I  =  1  TO  TOP 
NXT  =  I  +  GAP 
IF  A(I)  >  A ( NXT )  THEN 
SORTFLAG  =  0 

TEMP  =  A(  I) 

A ( I )  =  A ( NXT ) 

A (NXT)  -  TEMP 

END  IF 
NEXT  I 

NEXT 

NEXT 

LARGE (REP)  -  A ( N ) 

450  RETURN 

300  ( j  ■!•(!!!!!!!! M !!!!!!  i  !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!  ! 
510  !  Subroutine:  Calculate  slope  B  and  divide  LARGE  by  it. 

520  SXX  -  00 
330  SXY  -  0.  O 
540  FOR  I  -  1  TO  N 

SXX  -  SXX  ♦  NORM ( I >~2 
SXY  =  SXY  +  NORM ( I >  *  At  I) 

NEXT  I 

550  B  =*  SXY  /  SXX 

560  LARGE  (REP)  =*  LARGE  (REP)  /  B 

500  RETURN 


600  f  Subroutine:  Sort  tho  first  2/5  value*  of  array  LARGE 
610  FOR  I  «  1  TO  1  +  2  *  NREP  /  5 
LARG  -  LARGE t I > 

PSTN  -  I 

FOR  NXT  .  1  ♦  1  TO  NREP 

IF  LARGE (NXT)  >  LARG  THEN 
LARG  -  LARGE (NXT) 

PSTN  =  NXT 
END  IF 
NEXT  NXT 

LARGE (PSTN)  =  LARGE ( I ) 

LARGE! I)  “  LARG 
620  NEXT  I 
650  RETURN 

690  '  i  <  |  |  |  j  j  »  »  |  |  |  **»••'•*••  |  •!!!!!!!!!!!!!!!?!!!  ! 

800  !  Subroutine:  Calculate  alpha  values  and  print  results 
820  A05  *  LARGE  (NREP  /  20)  «*  LARGE (1  +  NREP  /  20) 

A05  =  AOS  /  2 

A 1 0  =  LARGE ( NREP  /  10)  +  LARGE ( 1  ♦  NREP  /  10) 

A10  =«  A10  /  2 

A20  =  LARGE (NREP  /  5)  +  LARGE ( 1  +  NREP  /  5) 

A20  =  A20  /  2 

A40  =  LARGE (2  *  NREP  /  5)  +  LARGE (1  +  2  *  NREP  /  5) 

A40  =  A40  /  2 

PRINT  #1  USING  A*i  N.  A05.  A10.  A2~.  A40 

830  RETURN 
999  Finish:  END 
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NOTATIONS  AND  TERMINOLOGY 


a 

Contrast 


d 

r 

e 

Effect 

e.v.n.o.s. 

E-spillover 

Expected  value  for 
normal  order 
statistics 

Half-normal  Plot 


Intended  error 
ranks 


Probability  of  rejecting  the  null  hypothesis  when  it  is 
true;  probability  of  making  a  Type  1  error. 

Difference  between  two  means  (the  high  and  low,  or 
positive  and  negative  conditions  of  a  balanced,  two-level 
factorial  design;  an  effect. 

Size  of  r  real  effects. 

Number  of  error  contrasts  (or  null  contrasts)  in  the 
results  of  a  2^  or  2^  ^  experiment. 

Size  of  the  mean  difference  or  contrast. 

Expected  values  for  normal  order  statistics. 

Error  contrasts  that  fall  at  ranks  where  real  effects  were 
introduced  prior  to  adding  an  error  component. 

Positions  on  the  normal  probability  scale  for  each  of 
n  items  in  rank  order. 


Plot  of  absolute  contrasts  on  half-normal  probability 
paper. 

Rank  position  of  a  value  in  a  set  of  ordered  values. 

Ranks  where  no  real  increments  were  added  before  real  and 
error  components  were  combined  and  reranked. 


NOTATIONS  AND  TERMINOLOGY  (cont’d) 


k  Number  of  experimental  effects.  In  unreplicated  2 

designs,  k  =  (n  -  1) . 

n  Number  of  independent  observations  in  an  experiment.  In 

an  urireplicated  2^  factorial  design,  n  =  k+1.  Also  n  = 

(r  +  e) . 

N[0,1]  Normally  distributed  population  with  a  mean  of  zero  and  a 

variance  of  one. 


Normal  plot  Ordered  signed  contrasts  plotted  on  normal  probability 

paper. 

PER  Probability  of  a  non-zero  "family"  error  rate,  dependent 

upon  number  of  contrasts  being  tested. 


Normal  probability  scale  value  at  rank  i. 


r 


r+ 


#R 

i ,  k 


\R 

i.k 


Half-normal  probability  scale  value  at  rank  i. 

Number  of  real  or  non-null  effects  in  the  set  of  k 
contrasts. 

Number  of  positive  real  effects  (r-  is  number  of 
negative  real  effects). 

Number  of  real  effects  that  occurred  at  rank  i  of 
the  ordered  set  of  k  contrasts  during  a  Monte  Carlo 
simulation  (of  5000  runs  in  this  report). 

Percent  real  effects  falling  at  rank  i  for  k 
effects . 
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NOTATIONS  AND  TERMINOLOGY  (cont'd) 

Real  effects  that  fall  at  ranks  intended  to  be  error 
contrasts. 

Population  sigma  or  standard  deviation,  a2  equals 
the  population  variance,  sometimes  referred  to  as  the 
error  variance. 

Estimation  of  the  population  a 
Final  estimation  of  the  population  a 

Standard  error  of  the  contrasts,  obta:  ed  by  multiplying 
the  estimated  population  sigma  by  the  square  root  of  four 
over  n. 

Standard  deviation  of  a  contrast  at  rank  i  of  k  contrasts 

Test  statistic  proposed  by  Daniel  used  to  determine  the 
probability  that  an  effect  of  a  certain  size  will  occur. 
Determines  location  of  guardrails. 

Slope  of  the  e  contrasts  on  a  normal  or  a  half-normal 
plot.  May  be  calculated  by  some  least  squares  regres¬ 
sion  technique.  The  slope  of  standardized  contrasts 
serves  is  used  to  estimate  the  population  sigma. 

Factorial  or  fractional  factorial  experiment  with  f 

factors  at  two  levels  each.  (f-p)  represents  a  1/2P 

fraction  of  a  2C  factorial.  [Note:  In  two  papers  in 

n 

Section  VII  on  “Relevant  Papers,"  the  symbols  2  and 
2  '  were  used  as  the  authors  had  used  them.  The  n  in 
that  case  is  f  as  used  in  this  report.] 


NOTATIONS  AND  TERMINOLOGY  (cont’d) 


Contrast  (l.e.,  mean  difference)  at  rank  1  of  k  ordered 
contrasts  for  normal  or  half-normal  data.  The  half-normal 
symbol  is  Indicated  by  the  prime. 

Standardized  contrasts,  obtained  by  dividing  raw  contrasts 
by  estimated  population  sigma  for  normal  and  half-normal 
data . 

Different  techniques  investigated  by  Zahn  for  estimating 
the  population  sigma  using  the  slope  of  the  standardized 
contrasts . 

Best  estimate  of  the  standard  error  of  contrasts  in  a  null 
experiment  with  k  contrasts,  with  a  =  integer  nearest 
0.638k. 

Estimated  value  of  order  statistics  at  rank  i  any  set  of  k 
values  for  normal  or  half-normal  data.  [Note:  The  k  is 
used  here  to  represent  the  number  of  contrasts  being 
plotted,  although  most  tables  of  e.v.n.o.s.  use  the  symbol 


